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ADAPTIVE  AND  NONADAPTIVE  TRAINING  TECHNOLOGY  FOR  SMALL  UNMANNED 
AERIAL  SYSTEM  EMPLOYMENT 

EXECUTIVE  SUMMARY 


Research  Requirement: 

The  U.S.  Army  Learning  Concept  for  2015  (ALC  2015),  TRADOC  Pamphlet  525-8-2 
(TRADOC,  2011)  lays  out  a  vision  for  how  the  Army  will  train  and  educate  Soldiers  to  execute 
full-spectrum  operations  in  an  era  of  persistent  conflict.  Learning  outside  the  classroom  will  play 
an  increasingly  key  role.  Innovative  learning  technologies  and  methods  will  be  required  to  make 
self-directed  learning  effective  and  efficient.  To  accomplish  this,  it  will  be  necessary  to  identify 
state-of-the-art  adaptive  training  capabilities,  and  develop  standards,  protocols,  and  guidance  on 
employing  these  capabilities  in  interactive  multimedia  instruction  (IMI)  modules.  To  this  end, 
this  report  describes  research  comparing  the  effectiveness  of  two  versions  of  a  training  system, 
one  adaptive  and  the  other  nonadaptive  (one-size-fits-all).  The  domain  addressed  was  small 
unmanned  aerial  system  (SUAS)  employment  aimed  at  leaders  at  company  and  below. 

Procedure: 

An  adaptive  training  prototype  system  was  developed  to  provide  instruction  and  decision¬ 
making  practice  for  the  domain  of  employment  of  SUAS  at  company  and  below.  In  particular, 
the  training  content  was  selected  to  provide  leaders  at  company  and  below  the  knowledge 
required  to  integrate  SUAS  operation  into  their  missions,  conduct  relevant  troop  leading 
procedures,  and  supervise  execution  of  SUAS  flights.  Adaptive  features  included  selection  of  up¬ 
front  instruction  based  on  pretest  performance,  branching  scenario-based  decision  making,  and 
within-scenario  remediation.  A  parallel  nonadaptive  training  system  was  also  developed. 
Twenty-four  Soldiers  were  provided  experience  with  one  of  the  two  versions  and  then  took  a 
post-training  test,  identical  to  the  pretest.  They  also  completed  questionnaires  on  their  opinions 
of  the  training  experience. 

Findings: 

The  posttest  results  were  inconclusive  with  respect  to  the  benefits  of  adaptation  on 
training  outcomes.  There  failed  to  be  a  significant  difference  in  learning  outcomes  between  the 
adaptive  and  the  nonadaptive  versions;  however,  there  were  learning  gains  with  both  systems, 
and  Soldiers  appraised  the  training  effectiveness  of  the  adaptive  system  to  be  superior  to  that  of 
the  nonadaptive  system. 

Utilization  and  Dissemination  of  Findings: 

Although  the  results  of  the  experiment  examining  learning  outcomes  after  adaptive 
training  vs.  nonadaptive  training  were  inconclusive,  these  results  can  only  speak  to  the  particular 
implementation  of  adaptive  training  tested.  They  do  not  necessarily  indicate  that  adaptive 
training,  in  general,  is  not  beneficial.  Our  implementation  may  have  failed  to  produce  the  desired 
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results  because  during  scenario-remediation  consisted  of  abstract  content  rather  than  specific 
remediation  related  to  the  scenarios.  Another  possibility  is  that  our  learning  outcome  measure 
may  have  been  insufficiently  sensitive  to  detect  differences  in  degree  of  learning. 

Both  versions  of  the  training  produced  evidence  of  learning  gains.  Therefore,  for 
purposes  of  actual  training,  there  is  the  potential  to  utilize  the  training  prototypes  and/or  to  invest 
in  their  further  development  for  purposes  of  training  SUAS  employment  for  leaders  at  company 
and  below.  Because  Soldiers  who  experienced  the  adaptive  training  prototype  rated  it  more 
positively  than  those  who  experienced  the  nonadaptive  prototype,  we  recommend  that  any 
further  development  utilize  that  version. 

Briefings  on  this  work  have  been  provided  to  the  Future  Force  Integration  Directorate, 
Fort  Bliss,  TX,  and  the  Maneuver  Center  of  Excellence  Directorate  of  Combat  Development,  the 
SUAS  School  at  Fort  Benning,  and  the  Institute  for  NCO  Professional  Development.  Results 
relating  to  this  work  have  been  presented  at  the  following  conferences:  Florida  Artificial 
Intelligence  and  Research  Society  (May  2010),  Applied  Human  Factors  and  Ergonomics  Society 
(July  2010),  Army  Science  Conference  (December  2010),  and  ITEC  2011  (May,  2011). 
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ADAPTIVE  AND  NONADAPTIVE  TRAINING  TECHNOLOGY  FOR  SMALL  UNMANNED 

AERIAL  SYSTEM  EMPLOYMENT 


Introduction 


The  U.S.  Army  Learning  Concept  for  2015  (ALC  2015),  TRADOC  Pamphlet  525-8-2 
(TRADOC,  2011)  lays  out  a  vision  for  how  the  Army  will  train  and  educate  Soldiers  to  execute 
full-spectrum  operations  in  an  era  of  persistent  conflict.  Learning  outside  the  classroom  will  play 
an  increasingly  key  role.  Innovative  learning  technologies  and  methods  will  be  required  to  make 
self-directed  learning  effective  and  efficient.  To  accomplish  this,  it  will  be  necessary  to  identify 
state-of-the-art  adaptive  training  capabilities,  and  develop  standards,  protocols,  and  guidance  on 
employing  these  capabilities  in  interactive  multimedia  instruction  (IMI)  modules.  To  this  end, 
this  report  describes  research  comparing  the  effectiveness  of  two  versions  of  a  training  system, 
one  adaptive  and  the  other  nonadaptive  (one-size-fits-all).  The  domain  addressed  was  small 
unmanned  aerial  system  (SUAS)  employment  aimed  at  leaders  at  company  and  below. 


There  is  strong  empirical  support  that  one-on-one  instruction  by  a  human  mentor  is 
superior  to  traditional  classroom-based  approaches  (e.g.,  Bausell,  Moody  &  Walzl,  1972;  Bloom, 
1984).  During  one-on-one  instruction,  the  mentor  has  the  ability  to  adapt  instructional  content 
and  style  as  deemed  most  suitable  to  the  individual  student.  A  classroom  teacher,  having  to  deal 
with  a  collection  of  individuals,  rarely  has  this  capability.  It  has  been  assumed  that  the  ability  of 
the  mentor  to  adapt  spontaneously  to  the  student  is  one  of  the  main  reasons  for  the  superiority  of 
one-on-one  instruction.  By  analogy,  it  has  been  assumed  that  for  technology-based  instruction, 
software  that  can  adapt  content  or  instructional  style  for  each  individual  student  should  be 
superior  to  one-size-fits  all  methods.  A  human  mentor  has  multiple  methods  of  adapting  to  the 
student  and  can  switch  easily  among  them;  however,  for  technology-based  instruction,  teaching 
methods  and  the  rules  for  when  to  apply  different  methods  need  to  be  decided  at  design  time. 

This  is  a  challenge  because  there  is  a  lack  of  causal  evidence  about  what  methods  are  most 
effective  under  different  conditions  (Durlach  &  Ray,  2011;  Ohlsson,  et  ah,  2007).  Many  different 
methods  have  been  implemented,  often  in  combination.  Both  Durlach  and  Ray  (2011)  and 
Vandervaetere,  Desmet,  and  Clarebout  (2011)  reviewed  adaptive  instructional  technologies,  and 
found  considerable  variation  in  system  design  and  sparse  data  related  to  empirical  effectiveness 
with  respect  to  enhancing  learning  outcomes. 

One  of  the  difficulties  in  evaluating  whether  automated  adaptive  techniques  have  the 
intended  benefits  on  learning  outcomes  compared  to  nonadaptive  technology-based  approaches 
has  been  the  lack  of  an  appropriate  baseline  condition  against  which  the  adaptive  system  can  be 
compared.  Comparing  learning  outcomes  from  such  a  system  with  an  entirely  different  form  of 
instruction  (e.g.,  classroom)  is  not  adequate  to  answer  this  question,  because  the  two  methods 
differ  in  many  ways  besides  the  software-based  adaptations.  Two  versions  of  the  same 
technology-based  instruction,  which  differ  only  in  the  adaptive  techniques  used,  is  what  is 
required.  The  purpose  of  the  research  presented  here  was  to  design  two  parallel  instructional 
prototypes  that  fit  this  requirement,  and  then  to  test  their  relative  effects  on  learning. 
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A  few  experiments  with  analogous  intentions  already  exist  in  the  literature  (Durlach  & 
Ray,  2011).  For  example,  Perrin,  Dargue,  and  Banks  (2003)  conducted  an  experiment  in  which 
they  compared  nonadaptive  training  on  export  control  rules  with  four  different  ways  of 
implementing  adaptive  training.  The  training  audience  consisted  of  company  employees  taking  a 
refresher  course,  having  had  training  two  years  previous.  In  the  Control  (nonadaptive)  condition, 
students  went  through  multimedia  instruction  and  practical  exercises  in  a  linear  fashion.  They 
were  given  intermittent  assessments  with  feedback;  but,  their  performance  on  these  assessments 
did  not  affect  their  path  through  the  material.  Learners  in  this  condition  could  initiate  review  of 
already  presented  information  at  their  own  discretion.  In  the  Mastery  Learning  condition, 
students  were  not  only  required  to  go  through  the  multimedia  instruction;  they  were  also  required 
to  review  material  related  to  items  they  got  incorrect  during  the  intennittent  assessments.  After 
this  review,  they  were  reassessed  before  being  able  to  progress.  A  second  adaptive  condition, 
Loop-Back  Remediation  was  similar  to  the  Mastery  Learning  condition,  but  with  an  additional 
adaptation:  incorrect  use  of  previously  mastered  material  led  to  required  remedial  review  of  that 
material.  So,  for  example,  if  an  error  was  made  on  an  assessment,  students  not  only  had  to 
review  material  relevant  to  the  unchosen  correct  option,  they  also  had  to  review  material  relevant 
to  the  chosen  incorrect  option.  The  third  adaptive  condition  was  the  Pretest  condition.  Learners 
in  this  condition  received  the  same  treatment  as  the  Loop-Back  Condition;  however,  prior  to 
training  they  took  a  knowledge  test,  which  allowed  them  to  skip  initial  multimedia  content 
related  to  items  passed  on  the  pretest.  Finally,  the  fourth  condition,  Advanced  Placement,  was 
also  identical  to  the  Loop-Back  condition,  except  that  the  first  phase  of  instruction  (covering  the 
first  seven  learning  objectives)  was  simply  omitted. 

Learners  in  the  Mastery  Learning  and  the  Loop-Back  conditions  performed  significantly 
better  on  a  training  posttest,  compared  to  the  Control  condition,  confirming  the  prediction  that 
adaptive  training  would  be  superior  to  nonadaptive  training.  However  the  scores  for  the  Pretest 
and  Advanced  Placement  conditions,  also  adaptive  treatments,  failed  to  be  statistically  superior 
to  the  Control  condition.  Thus,  allowing  students  to  skip  instruction  based  on  either  assumed 
knowledge  (Advanced  Placement)  or  prior  testing  results  (Pretest),  turned  out  to  interfere  with 
the  beneficial  effects  of  the  loop-back  strategy.  This  was  likely  because  the  loop-back  strategy 
was  only  applied  to  presented  material.  The  Advanced  Placement  and  Pretest  conditions  skipped 
some  material  (and  assessments).  Consequently,  there  was  no  opportunity  to  detect  knowledge 
gaps  that  were  attributed  mistakenly  as  known  until  the  posttest.  For  example,  for  the  Pretest 
condition,  lucky  correct  guesses  on  the  pretest  would  allow  the  student  to  skip  relevant  didactic 
instruction  (and  its  related  assessment).  Material  on  the  same  topic  would  not  be  tested  again 
until  the  posttest.  Thus,  for  the  Advanced  Placement  and  Pretest  conditions,  un-mastered 
learning  objectives  incorrectly  assumed  as  mastered  were  never  remediated. 

In  the  present  research,  the  adaptive  condition  allowed  students  to  skip  didactic 
instruction  associated  with  pretest  knowledge  components  answered  correctly,  just  as  in  Perrin, 
Dargue,  and  Banks’  (2003)  Pretest  condition;  however,  they  subsequently  went  on  to  complete 
scenario-based  exercises.  Substandard  perfonnance  during  these  exercises  induced  remediation. 
Thus,  these  scenario-based  exercises  could  catch  any  knowledge  that  had  been  incorrectly 
inferred  as  known  (as  a  result  of  lucky  guessing  on  the  pretest).  In  the  nonadaptive  condition, 
students  also  completed  the  pretest,  but  only  for  purposes  of  evaluation.  They  were  required  to 
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review  all  the  didactic  instruction  regardless  of  their  pretest  performance.  They  then  went  on  to 
complete  the  scenario-based  exercises.  All  trainees  completed  a  posttest,  which  was  identical  to 
the  pretest.  The  aim  was  to  evaluate  whether  greater  pretest  to  posttest  learning  gains  would  be 
observed  in  the  adaptive  condition  compared  with  the  nonadaptive  condition,  or  whether 
equivalent  gains  would  be  observed  but  in  less  time  in  the  adaptive  condition  compared  with  the 
nonadaptive  condition. 


Knowledge  Domain 

The  domain  covered  by  the  training  prototype  was  SUAS  employment  for  Army  leaders 
at  company  and  below.  Unmanned  aerial  systems  are  considered  ‘small’  if  they  are  man-portable 
and  their  employment  does  not  require  an  established  infrastructure  (such  as  a  runway  or 
airport).  It  has  been  envisioned  that  SUAS  will  be  operated  by  privates  through  sergeants  (Office 
of  the  Secretary  of  Defense,  2002);  but,  more  senior  personnel  will  decide  when  and  how  to 
employ  SUAS.  These  more  senior  members  of  the  operator’s  unit  will  be  the  ones  who  plan  and 
supervise  SUAS  missions.  Consequently,  company  commanders,  platoon  leaders  and  platoon 
sergeants  need  training  on  the  factors  they  ought  to  consider  in  the  use  of  SUAS,  including 
potential  risks  and  benefits,  and  coordination  with  other  users  of  the  air  space  (Durlach,  2007). 
Thus,  the  focus  of  the  training  was  on  procedures  of  SUAS  employment  and  principles  of 
operation  at  company  or  troop  level  as  part  of  a  battalion  or  squadron  mission.  This  includes 
synchronizing  the  use  of  the  SUAS  into  the  overall  tactical  plan  in  accordance  with  higher 
commander’s  intent,  taking  into  account  Mission,  Enemy,  Time,  Terrain/Weather,  Troops 
Available,  and  Civil  Considerations  (METT-TC).  It  also  includes  understanding  air  space 
coordination  procedures:  integration  and  synchronization  of  the  SUAS  mission  within  the 
constraints  posed  by  other  users  of  the  air  space. 

Training  Audience 

The  training  audience  was  intended  to  include  Maneuver  Company  Commanders  (CO), 
Executive  Officers  (XO),  Platoon  Leaders  (PL),  Robotics  noncommissioned  officers  (NCO)  or 
SUAS  Team  Leaders  (E6/E7);  in  other  words,  any  personnel  in  charge  of  planning,  preparing 
for,  and/or  supervising  the  execution  of  SUAS  missions. 

Domain  Analysis 

The  domain  analysis  was  conducted  by  two  Boeing  analysts.  They  gathered  relevant 
reference  materials  and  marshaled  their  own  military  experience  to  conduct  a  task  analysis.  Two 
primary  sources  were:  Leader's  Guide  to  A2C2  at  Brigade  and  Below  (Center  for  Army  Lessons 
Learned,  2005),  and  a  task  analysis  conducted  under  the  Future  Combat  System  project  for  the 
task  Synchronize  Planning  of  Air-Ground  Operations  for  Company  and  Platoon.  A 
comprehensive  list  of  references  is  provided  in  Appendix  A.  The  goal  of  the  task  analysis  was  to 
determine  what  the  training  audience  must  know  in  order  to  employ  a  SUAS,  and  then  to  create 
tactical  scenarios  employing  SUAS  assets  requiring  decision-making  based  on  that  knowledge. 
Decision  points  in  the  scenarios  were  designed  to  tap  knowledge  of  generic  SUAS  tactics, 
techniques,  and  procedures  (TTPs);  technical  infonnation  concerning  the  capabilities  of  a 
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specific  SUAS  would  be  available  in  reference  material  if  needed  to  make  a  decision.  So  for 
example,  a  decision  might  require  knowledge  that  weather  considerations  are  important;  the 
specific  weather  conditions  under  which  the  particular  SUAS  could  or  could  not  fly  were 
available  as  reference  materials  and  were  not  part  of  what  was  to  be  learned. 

The  task  analysis  resulted  in  nine  tenninal  learning  objectives  (TLOs).  These  are  listed 
below,  with  their  associated  enabling  learning  objectives  (ELOs). 

1 .  SUAS  Role  in  Operations:  Understand  the  role  of  the  SUAS  in  the  overall  tactical 
plan/operation. 

•  Attack,  Defense,  Security,  Civil  Operations 

•  Reconnaissance  and  Surveillance 

•  Target  Acquisition  and  Battle  Damage  Assessment 

•  Security 

2.  Tactical  Control  Measures:  Understand  common  company  and  battalion  tactical  control 
measures. 

•  Named  area  of  interest 

•  Axis  of  advance 

•  Battle  position 

•  Engagement  area 

•  Attack  by  fire  position 

•  Observation  post 

3.  Procedural  Airspace  Controls:  Understand  the  three  types  of  procedural  airspace  separation. 

•  Lateral  separation 

•  Temporal  separation 

•  Vertical  separation 

4.  SUAS  Mission  requests:  understand  the  two  types  of  mission  requests  and  the  three  types  of 
missions. 

•  Planned 

•  Immediate 

•  Dynamic 

5.  Importance  of  airspace  coordination:  Understand  the  importance  of  airspace  coordination. 

•  Other  unmanned  aerial  vehicles 

•  Manned  aircraft 

•  Direct  and  indirect  fires 

6.  Importance  of  coordination  to  support  situational  awareness:  Understand  the  importance  of 
coordinating  with  other  units  potentially  affected  by  SUAS  operations. 

•  Other  units  (higher,  lower,  adjacent) 

•  Observations  and  reporting 

•  Battle  Damage  Assessment 
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7.  Understand  the  airspace  environment:  Understand  the  risks  in  the  airspace  environment. 

•  Powerlines  and  manmade  structures 

•  Communications  interference 

•  Low  altitude  civilian  aircraft 

•  Close  combat  attack  aviation 

•  Terrain  and  line  of  sight 

•  Positive  control  and  data  links 

•  Enemy  air  defense 

•  Close  air  support 

•  Weather 

8.  Procedural  controls:  Understand  the  procedural  control  measures  used  in  the  airspace. 

•  Formal  and  informal  airspace  coordination  areas 

•  Restrictive  fire  areas  and  lines,  no  fire  areas 

•  Common  reference  system 

•  Air  control  point 

•  Air  corridors 

•  Coordinating  altitude 

•  Restricted  operating  zones  and  areas 

•  Minimum  risk  corridor 

9.  Brigade  responsibilities:  Understand  the  roles  and  responsibilities  for  airspace  coordination  at 
the  brigade  and  battalion  levels. 

•  Mission  authorization 

•  Battalion  staff  responsibilities 

•  Company  responsibilities 

•  Emergency  procedures 

Appendix  B  provides  the  linkage  between  these  learning  objectives  and  the  source  materials 
from  which  they  were  derived. 


Scenario  Development 

The  scenarios  developed  for  the  training  prototype  consisted  of  Plan,  Prepare,  and 
Execute  phases.  The  goal  in  scenario  development  was  to  identify  key  decision  points  within 
each  phase,  which  would  require  knowledge  of  the  learning  objectives.  Generic  scenario  decision 
points  for  each  of  these  phases  are  listed  in  Table  1.  These  decision  points  were  used  as  a 
framework  to  develop  two  scenarios,  an  offense  and  a  defense  scenario. 
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Table  1.  Generic  Decision  Poin  ts  for  each  Phase  of  a  Scenario _ 

PLAN 

•  Decision  Point  #1  -  Analysis  of  Commander’s  Intent  in  relation  to  SUAS  operations 

•  Decision  Point  #2  -  Development  of  Flight  Plan 

•  Decision  Point  #3  -  Development  of  Actions  on  Objective 

•  Decision  Point  #4  -  Development  of  Launch/Recovery/Next  Operations  Transition 

PREPARE 

•  Decision  Point  #5  -  Rehearsal 

•  Decision  Point  #6  -  Communications  Checks/Contingency  Plans  for  Positive  Control 

•  Decision  Point  #7  -  Emergency  Destruction  Contingency/Change  of  Mission 

EXECUTE 

•  Decision  Point  #8  -  Synchronization  of  Fires  and  Maneuvers 

•  Decision  Point  #9  -  Support  Changes  to  maneuver  plan 

•  Decision  Point  #10  -  Avoid  &  Reduce  SUAS  fratricide 

•  Decision  Point  #1 1  -  Support  Attack  -  High  Payoff  Targets  -  Target  Acquisition 


The  two  Boeing  employees  who  conducted  the  task  analysis  also  developed  the 
scenarios.  Both  scenarios  began  by  presenting  the  student  with  a  description  of  the  mission 
situation,  with  information  relevant  to  commander’s  intent.  For  example,  the  offense  scenario 
began  with  presentation  of  a  tactical  map  and  the  following  description: 

SITUATION:  Tm  A/1-52  Inf:  Mission  is  to  attack  and  destroy  an  enemy  platoon 
on  the  outskirts  of  a  small  village  during  limited  visibility  conditions  (e.g.  night). 

Bn  Cdr  Intent:  The  destruction  of  the  enemy  positioned  (OBJ  CROSS)  near  the 
village  is  vital  to  the  overall  battalion  mission.  The  momentum  of  the  attack  to 
OBJ  JAB  (Bn's  Objective)  must  not  be  slowed  by  enemy  fires  from  OBJ  CROSS. 

End  State:  After  destruction  of  the  enemy,  TM  A  establishes  a  blocking  position 
on  the  avenue  of  approach  passing  through  the  village.  The  Bn  Cdr  directs  that 
reconnaissance  plans  be  focused  on  identification  of  enemy  anti-tank  guided 
missiles  positions. 

Each  decision  point  in  the  scenario  involved  a  situation  update  and  a  request  for  a 
decision.  An  ideal  choice  (according  to  existing  doctrine)  was  decided  during  initial  scenario 
development.  Once  this  scenario  “trunk”  was  developed,  the  scenario  developers  went  back  and 
created  three  alternative  choices  for  each  decision  point.  The  aim  was  that  each  decision  point 
have  an  optimal  choice,  two  acceptable  choices,  and  one  unacceptable  choice.  For  the  adaptive 
version  of  the  prototype,  the  acceptable  choices  allowed  the  student  to  deviate  from  the  trunk  and 
enter  a  scenario  branch.  For  decision  points  on  the  branches,  selecting  an  acceptable  choice 
continued  on  the  branch  or  returned  the  student  to  the  trunk.  This  ability  to  return  to  the  trunk 
allowed  the  degree  of  branching  to  be  kept  manageable.  Figure  1  illustrates  the  branching 
structure  for  the  offense  scenario. 
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Figure  1.  Illustration  of  the  branching  structure  of  the  adaptive  offense  scenario.  If  the 
student  chose  an  unacceptable  option  at  a  decision  point,  they  were  given  feedback  and 
were  required  to  choose  a  different  option.  In  the  nonadaptive  condition,  students  were 
kept  on  the  optimal  path  (trunk),  regardless  of  which  decision  they  chose. 


Table  2  provides  an  example  decision  point  (the  first  decision  point  in  the  offense 
scenario).  For  each  option,  scenario  developers  also  created  a  feedback  message  for  each  choice, 
also  illustrated  in  Table  2.  Feedback  was  accompanied  by  a  green,  yellow,  or  red  graphic  for 
optimal,  acceptable,  and  unacceptable  decisions,  respectively. 

For  each  decision,  scenario  authors  created  three  levels  of  hints,  also  illustrated  in  Table 
2.  While  making  a  decision  a  student  (in  either  condition)  could  request  the  first  hint,  “for  free.” 
Requests  for  additional  hints  incurred  a  cost  to  their  scores. 
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Table  2.  Textual  Elements  Contributing  to  a  Decision  Point,  Using  the  First  Decision  Point  in 
the  Offense  Scenario. Decision  Points  also  Included  a  Graphic,  Usually  a  Tactical  Map 


SITUATION 

Tm  A/1-52  Inf:  Mission  is  to  attack  and  destroy  an  enemy  platoon  on  the  outskirts  of  a  small 
village  during  limited  visibility  conditions  (e.g.  night). 

Bn  Cdr  Intent:  The  destruction  of  the  enemy  positioned  (OBJ  CROSS)  near  the  village  is  vital  to 
the  overall  battalion  mission.  The  momentum  of  the  attack  to  OBJ  JAB  (Bn's  Objective)  must 
not  be  slowed  by  enemy  fires  from  OBJ  CROSS.  End  State:  After  destruction  of  the  enemy,  TM 

A  establishes  a  blocking  position  on  the  avenue  of  approach  passing  through  the  village.  The  Bn 
Cdr  directs  that  reconnaissance  plans  be  focused  on  identification  of  enemy  anti-tank  guided 
missiles  positions. 

QUESTION 

You  are  the  Company  Cdr/XO.  Based  on  Bn  Cdr's  intent  for  your  assigned  mission,  what  should 
be  the  primary  task  of  the  SUAS  Team  to  support  the  attack  at  OBJ  CROSS? 

OPTION  1  (Optimal) 

Primary  Intent  is  Location  of  Enemy  Weapons  Locations 

Feedback  if  Option  1  selected:  The  CDR's  INTENT  states  the  enemv  must  be  destroved. 
Enemy  crew-served  weapons  are  the  key  to  the  enemy  defense.  Knowledge  of  their  location  and 
destruction  will  neutralize  the  enemy's  capability  to  defend.  The  Commander  stresses  this 
critical  need  in  his  reconnaissance  guidance.  SUAS  execute  the  company  reconnaissance  plan. 

OPTION  2  (leads  to  a  branch) 

Primary  Intent  is  Identification  of  Enemy  Activity 

Feedback  if  Option  2  selected:  Enemv  activity  often  is  indicative  of  enemv  intent  and 
capabilities;  the  responsiveness  to  the  Cdr's  Intent  with  respect  to  enemy  locations  should  be 
considered. 

OPTION  3  (leads  to  a  branch) 

Primary  Intent  is  Location  of  Enemy  Obstacles 

Feedback  if  Option  3  selected:  Knowledge  of  location  of  enemv  obstacles  aids  in 
determining  an  enemy  defensive  plan.  Enemy  fires  habitually  cover  obstacles  so  friendly  forces 
are  channeled  into  kill  zones.  Knowledge  of  obstacle  locations  is  an  indirect  method  of  locating 
enemy  weapons.  After  location  of  enemy  weapons,  location  of  obstacles  is  a  secondary  priority 
for  offensive  reconnaissance. 

OPTION  4  (unacceptable) 

Primary  Intent  is  ID  of  Enemy  Command  Post 

Feedback  if  Option  4  selected:  CP  location  is  useful  intelligence.  But  it  is  not  relevant 
intelligence  needed  for  destruction  of  enemy  weapons.  Back  up  and  choose  another. 

HINTS 

Level  1 :  Who  is  the  primary  unit  who  needs  the  RSTA  information  and  what  information  do  they 
need? 

Level  2:  Why  is  the  RSTA  information  important  to  the  Battalion  and  Company  missions? 

Level  3:  Who  turns  the  sensor  data  into  actionable  intelligence? 
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The  scenario  developers  determined  how  selection  of  any  given  decision  choice  should 
update  student  scores.  Student  scores  were  represented  in  an  array  with  one  number  for  each 
ELO.  This  array  is  the  student  model.  The  student  model  is  updated  immediately  after  each 
problem-solving  decision,  and  therefore  represents,  at  any  point  in  time  during  the  training,  the 
student’s  estimated  level  of  mastery  on  each  of  the  ELOs.  A  one-to-many  relationship  was 
allowed  between  a  decision  choice  and  the  ELO  updates.  In  other  words,  a  particular  student 
response  could  cause  updating  of  more  than  one  ELO.  For  optimal  choices,  all  related  ELOs 
were  impacted  positively;  and  for  unacceptable  choices,  all  the  related  ELOs  were  impacted 
negatively.  For  acceptable,  but  non-optimal  choices,  some  ELOs  could  be  impacted  positively, 
while  others  could  be  impacted  negatively.  The  degree  of  impact  each  choice  had  on  its  related 
ELO  scores  was  not  necessarily  equal.  The  positive  or  negative  increment  made  to  a  particular 
ELO  depended  on  how  strongly  the  decision  point  was  relevant  to  that  ELO,  according  to  the 
scenario  author. 


Implementation  of  the  Adaptive  Prototype 

The  adaptive  training  prototype  was  developed  by  Boeing,  using  a  combination  of 
commercial-off-the-shelf  (e.g.,  Microsoft  Excel),  open  source  (e.g.,  AuthorPOINT,  Mind  On 
Sight  Solo),  and  Boeing  in-house  tools.  The  prototype  is  fully  conformant  and  compliant  to 
SCORM  2004,  and  can  be  run  in  a  web-based  learning  management  system  (LMS).  The  LMS 
used  for  prototype  testing  was  an  open  source  system  called  ILIAS  (see  http://www-ilias.cea.fr). 
Figure  2  gives  an  overview  of  the  analysis,  design  and  implementation  processes,  indicating 
where  different  tools  were  used  in  the  various  processes. 

Scenario  Flow 

Scenario  flow  is  fairly  straight  forward  for  a  student  who  makes  only  optimal  or 
acceptable  choices  and  whose  performance  remains  above  threshold.  As  previously  described, 
each  scenario  begins  by  presenting  the  student  with  a  description  of  the  mission  situation,  with 
information  relevant  to  commander’s  intent.  This  is  followed  by  a  series  of  decision  points, 
where  each  decision  point  provides  a  situation  update  and  a  request  for  a  decision  choice  from 
among  four  options  (one  optimal,  two  acceptable,  and  one  unacceptable).  Before  deciding,  the 
student  can  ask  to  look  at  reference  material  (including  a  glossary,  the  scenario  mission 
statement,  and  SUAS  specifications),  or  request  a  hint.  Asking  for  one  hint  has  no  effect; 
however  asking  for  a  second  hint  halves  the  positive  ELO  credits  the  student  gets  for  a 
subsequent  acceptable  choice.  If  the  student  requests  all  three  of  the  available  hints,  they  get  no 
credit  for  a  subsequent  choice.  They  also  get  a  “strike”  against  the  relevant  TLO(s). 

If  the  student  has  selected  the  optimal  or  an  acceptable  decision  option,  the  system 
updates  the  student  model  (modulated  by  the  number  of  hints  requested)  and  provides  the 
appropriate  feedback.  The  student  then  advances  to  the  next  decision  point.  This  continues  until 
the  student  completes  the  scenario,  at  which  point  they  are  presented  with  a  performance 
summary.  Figure  3  illustrates  the  information  provided  in  the  performance  summary. 
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Figure  2.  Overview  of  analysis,  design,  and  implementation  process. 
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•  Out  of  19  decisions,  X  (YY%)  of  your  choices  were  Optimal 
decisions. 

•  You  asked  for  help  Z  times. 

•  You  only  made  W  "Doomsday"  decisions. 

•  You  seem  to  have  a  very  good  understanding  of  the  following 
objectives. 

-  List  of  ELOS  with  scores  above  0.8 


•  You  scored  low  on  the  following  objectives. 

-  List  of  ELOS  with  scores  below  0.8 


Figure  3.  Schematized  version  of  the  performance  summary  provided  after  completion  of  a 
scenario. 


If  the  student  selects  an  unacceptable  option,  the  system  updates  the  student  model, 
provides  feedback,  instructs  the  student  to  make  a  different  choice,  and  presents  the  situation  and 
options  again.  The  student  also  incurs  a  “strike”  against  the  relevant  TLO(s).  A  schematic 
illustrating  this  flow  is  shown  in  Figure  4. 

Remediation  during  Scenarios 

If  a  student  accumulates  three  strikes  against  any  TLO,  the  scenario  terminates,  and  the 
student  is  provided  didactic  remediation  on  ELOs  related  to  their  poor  perfonnance.  As 
illustrated  in  Figure  5,  this  is  where  the  phases  (Plan,  Prepare,  and  Execute)  play  a  role.  If  the 
scenario  is  tenninated  for  remediation,  then  subsequent  to  remediation,  the  student  re-enters  the 
scenario  at  the  beginning  of  the  phase  they  were  in  when  they  exited.  In  addition,  at  the  end  of 
each  phase,  the  system  evaluates  the  state  of  the  student  model,  and  if  perfonnance  is  below 
criterion  on  any  ELO,  the  scenario  is  aborted  and  the  student  receives  remediation.  Like  for  the 
strikes,  after  remediation,  the  student  returns  to  the  start  of  the  mission  phase  they  had  just  failed. 
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Figure  4.  Illustration  of  scenario  flow  when  the  student  selects  an  unacceptable  (No-Go) 
decision. 


Remediation  materials  were  taken  directly  from  reference  sources  used  in  the  original 
task  analysis  (Appendix  A).  Each  screen  stated  the  ELO,  cited  the  relevant  references,  and 
included  information  from  the  references,  along  with  some  relevant  imagery.  This  was  a 
potential  weakness  of  the  system,  because  the  reference  materials  were  not  designed  for 
instructional  remediation,  and  did  not  refer  specifically  to  the  scenario  decisions  that  the  student 
had  erred  on.  Rather,  the  content  was  relatively  abstract,  providing  excerpts  from  handbooks  and 
field  manuals.  Were  more  development  on  the  prototype  to  occur,  it  is  recommended  that  the 
remediation  materials  link  back  more  specifically  to  the  particular  kinds  of  errors  students 
committed,  with  reference  to  the  scenario  context.  Such  error-sensitive  feedback  has  been  shown 
to  facilitate  learning  (Durlach  &  Ray,  2011). 

Testing-In  to  Upfront  Instruction 

The  same  materials  that  were  prepared  for  during-scenario  remediation  were  also  used  for 
upfront  instruction.  The  scenario  developers  created  a  knowledge  test  (multiple  choice)  covering 
the  ELOs.  Performance  on  this  knowledge  test  was  used  to  tailor  upfront  instruction.  Only 
instruction  related  to  ELOs  for  questions  missed  was  presented.  The  student  began  the  first 
scenario  after  reviewing  this  material.  In  an  authentic  training  situation,  students  would  receive 
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Figure  5.  Overview  of  the  flow  of  the  adaptive  training  system.  Students  are  assumed  to  have 
some  incoming  knowledge,  and  to  gain  knowledge  by  reviewing  the  instruction  (p  =  0.7).  This 
value  is  decremented  (on  an  ELO  basis)  according  to  errors  committed  on  the  pretest  or  less  than 
optimal  choices  selected  during  decision  making.  Students  initially  take  a  pretest.  Instruction  is 
provided  on  any  ELOs  for  which  errors  were  made.  After  this,  the  student  begins  the  decision 
making  phase  of  training.  This  scenario-based  phase  can  be  interrupted  with  remedial  instruction 
if  student  model  scores  on  any  ELO  fall  below  the  criterion  threshold  (p  =  0.6)  or  if  three  strikes 
accumulate.  This  remediation  is  followed  by  a  return  to  the  decision  making  context,  at  the  start 
of  the  phase  that  had  been  interrupted,  p  =  0.8  or  above  on  all  ELOs  was  set  as  the  pass  criterion. 
After  completing  a  scenario,  if  students  did  not  meet  this  criterion,  they  were  required  to  repeat 
the  scenario  (for  a  maximum  of  three  repetitions). 


corrective  feedback  on  their  test  answers;  however,  for  purposes  of  evaluating  the  adaptive 
prototype  vs.  a  nonadaptive  version,  corrective  feedback  was  not  given.  Students  did  receive  a 
performance  summary  (percent  correct). 

The  original  version  of  the  pretest  included  42  questions  and  covered  all  the  ELOs 
thoroughly.  Pilot  testing  indicated  that  Soldiers  took  almost  an  hour  to  complete  this  test, 
however.  Rather  than  devoting  so  much  time  to  pretesting,  the  test  was  reviewed  by  the  scenario 
authors  and  shortened  to  22  questions.  This  meant  that  a  few  of  the  ELOs  did  not  have  an 
associated  question,  and  no  ELOs  were  covered  by  multiple  questions.  This  compromise  reduced 
the  test’s  sensitivity,  but  took  about  half  the  time  to  complete. 
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Parametric  Decisions 


Implementation  of  the  prototype  required  several  parametric  decisions,  which  were 
somewhat  arbitrary  and  largely  based  on  developer  intuition.  The  scenario  developers  decided 
the  weights  used  to  update  ELO  scores  for  each  decision  option.  The  system  developers  decided 
that  each  decision  would  have  three  hints  and  how  hint  requests  would  affect  updating  the 
student  model  (e.g.,  that  full  credit  would  be  given  with  one  hint,  half  credit  with  two  hints,  and 
no  credit  with  three  hints).  They  decided  that  ELO  scores  could  range  from  -  1  to  +  1.  They  set 
criteria  for  when  to  abort  a  scenario  and  send  the  student  to  remediation  (e.g.,  three  strikes  or  any 
ELO  below  0.6  at  the  end  of  a  phase).  Another  parameter  selected  was  how  to  update  the  student 
model  after  students  reviewed  didactic  material.  Assuming  that  students  gained  some  knowledge 
by  reviewing  the  upfront  instruction  or  remedial  slides,  the  ELO  scores  associated  with  that 
material  were  updated  to  0.7  after  review.  Finally,  0.8  or  above  on  all  ELOs  was  set  as  the  pass 
criterion.  After  completing  a  scenario,  if  students  did  not  meet  this  criterion,  they  were  required 
to  repeat  the  scenario  (for  a  maximum  of  three  repetitions). 

Subject  Matter  Expert  Feedback 

Instructors  and  other  available  personnel  associated  with  the  Raven  Operator  School  at 
Fort  Benning,  Georgia  generously  spent  time  with  the  development  team  over  a  two-day  period, 
examining  an  early  version  of  the  prototype  and  providing  feedback.  This  included  the  chief  of 
an  in-theater  mobile  training  team,  experienced  in  SUAS  unit  operations.  Our  goals  were  to 
assess  system  usability,  obtain  expert  feedback  on  our  content,  and  gain  additional  content 
knowledge  about  current  TTPs  from  hands-on  users  of  SUAS.  Raven  school  personnel  took  the 
knowledge  test  (original  version),  completed  one  or  both  scenarios  (depending  on  their  available 
time),  reviewed  a  sample  of  the  instructional  materials,  and  provided  feedback  on  all  these 
components.  Participants  were  generally  approving  of  the  prototype,  concurred  with  our  TLOs, 
and  thought  that  it  would  be  a  valuable  component  of  a  leader’s  course  on  SUAS  employment, 
were  one  to  be  developed.  We  learned  from  them  about  current  TTPs,  which  did  not  necessarily 
concur  with  written  doctrine.  We  also  learned  about  training  deficiencies  that  instructors  thought 
existed  in  their  current  curricula,  and  areas  that  were  unaddressed  due  to  the  inability  of  the 
school  to  keep  pace  with  the  evolution  of  SUAS  employment  in-theater.  We  subsequently 
injected  this  infonnation  into  the  system.  Particularly  useful  was  their  input  on  current  but  non- 
doctrinal  TTPs.  This  was  used  to  help  fashion  the  acceptable,  but  non-doctrinal  decision  options. 


Implementation  of  the  Nonadaptive  Prototype 

Were  we  developing  a  nonadaptive  prototype  for  actual  training  purposes,  we  would  have 
developed  an  event  flow  as  depicted  on  the  left  side  of  Figure  6;  however,  we  were  designing  the 
nonadaptive  version  in  order  to  conduct  an  experiment,  which  would  compare  training  outcomes 
for  the  adaptive  and  nonadaptive  versions.  To  that  end,  certain  modifications  were  made  to  the 
authentic  flow.  This  is  depicted  on  the  left  side  of  Figure  6.  Specifically,  an  authentic  version, 
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would  present  a  block  of  instruction,  followed  by  check  on  learning  questions  with  corrective 
feedback,  followed  by  the  practical  exercises  (scenarios).  The  problem  with  this  is  that,  for  the 
experiment,  we  wanted  to  give  all  students  the  knowledge  test  as  a  pretest  prior  to  any 
instruction.  This  would  allow  us  to  examine  knowledge  gains  by  comparing  perfonnance  on  a 
pretest  with  that  of  a  posttest.  If  we  used  the  authentic  flow,  students  in  the  nonadaptive 
condition  would  have  to  complete  both  the  pretest  and  the  check  on  learning  questions.  Not  only 
would  this  take  more  time,  the  check  on  learning  questions  would  give  the  nonadaptive  condition 
additional  test  questions.  It  is  known  that  testing  improves  retention  (Carpenter  &  DeLosh,  2005; 
Komell,  Hays,  &  Bjork,  2009;  Pyc  &  Rawson,  2010),  and  so  we  did  not  want  to  give  different 
amounts  of  testing  in  the  two  conditions.  Consequently,  we  implemented  the  flow  illustrated  on 
the  right  of  Figure  6.  Students  would  first  take  the  pretest,  then  review  all  the  didactic  instruction 
(regardless  of  pretest  performance),  and  then  go  on  to  complete  the  scenarios. 

Scenario  flow  for  the  nonadaptive  version  of  the  training  is  illustrated  in  Figure  7.  All 
aspects  of  scenario  decision  making  were  the  same  in  the  adaptive  and  nonadaptive  prototypes, 
except  that  the  nonadaptive  scenario  did  not  branch,  and  it  did  not  adaptively  remediate. 
Regardless  of  whether  the  optimal  or  one  of  the  acceptable  answers  was  chosen,  feedback  was 
given,  followed  by  the  next  situation  on  the  trunk.  If  the  unacceptable  answer  was  selected,  like 
for  the  adaptive  condition,  the  student  would  be  given  feedback  and  instructed  to  select  another 
option.  The  nonadaptive  version  provided  the  same  access  to  hints  and  reference  materials  as  did 
the  adaptive  version.  At  the  end  of  a  scenario,  students  had  the  option  to  review  the  instructional 
materials  relevant  to  their  weaknesses  (pointed  out  in  the  performance  summary),  by  selecting 
items  from  a  list. 
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Authentic  Nonadaptive  Flow 


Experimental  Nonadaptive 
Flow 


Figure  6.  Comparison  of  a  potential  training  flow  for  an  authentic  nonadaptive  training 
system  (left)  and  the  actual  flow  of  the  nonadaptive  training  system  used  for  experimentation 
(right). 
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Nonapdaptive  Scenario  Flow 


Situation 


(  choices 


Figure  7.  Illustration  of  scenario  flow  during  decision  making  in  the  nonadaptive  version. 
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Evaluating  the  Adaptive  vs.  the  Nonadaptive  Training  Prototypes 


Participants 

In  order  to  compare  the  effectiveness  of  the  adaptive  and  nonadaptive  versions,  we 
sought  the  participation  of  Soldiers  at  two  different  Anny  posts.  Soldiers  all  gave  informed 
consent,  and  were  randomly  assigned  to  conditions.  We  requested  12  each:  company 
commanders  (Captains),  platoon  leaders  (Lieutenants),  and  noncommissioned  officers  (Staff 
Sergeants  or  Sergeants  First  Class),  from  maneuver  units  (infantry  and  armor).  The  actual 
Soldiers  tested  were  two  Captains,  12  Lieutenants,  four  Staff  Sergeants,  and  13  Sergeants  First 
Class,  and  came  from  nine  different  branches  (e.g.,  infantry,  artillery,  military  police,  etc.). 
Overall,  16  Soldiers  completed  the  adaptive  version,  and  15  the  nonadaptive  version;  however, 
data  from  seven  of  the  participants  were  excluded  from  the  analysis  due  to  malfunctioning  of  the 
software  during  their  training  sessions.  This  resulted  in  data  for  13  Soldiers  in  the  adaptive 
version,  and  1 1  in  the  nonadaptive  version. 

Equipment  and  materials 

Four  laptop  computers  were  used  for  the  training.  Two  of  the  laptops  were  loaded  with 
the  adaptive  version  of  the  training  software  and  two  with  the  nonadaptive  version.  The  research 
was  conducted  in  a  well-lit  room  with  a  large  table  with  two  laptops  on  either  side.  Up  to  four 
Soldiers,  one  per  laptop,  could  participate  simultaneously.  Soldiers  were  asked  to  complete  one 
paper  questionnaire  prior  to  training,  which  asked  about  experience  and  background,  and  another 
after  training,  which  asked  about  reactions  to  and  opinions  of  the  training.  The  pretest  and 
posttest  were  completed  on  the  laptops. 

Procedure 

Each  Soldier  participated  for  about  half  a  day  (9:00  -  12:30  or  13:30  -17:00).  Upon 
arriving  at  the  testing  location,  Soldiers  were  randomly  assigned  to  a  computer,  and  asked  to 
look  over  information  about  the  research,  including  infonnation  about  infonned  consent  and 
their  rights  of  privacy.  If  they  agreed  to  participate  (all  did)  they  signed  an  informed  consent 
form  and  were  assigned  a  participant  code,  which  would  be  used  to  identify  their  data.  After 
completing  the  background  questionnaire,  the  Soldiers  were  shown  how  to  log  into  the  training 
system  using  their  code,  and  then  completed  training  at  their  own  pace.  They  were  told  to  ask 
questions  if  they  ran  into  any  difficulties  or  had  any  confusion  about  the  system.  They  were  also 
told  our  estimate  of  how  long  the  session  would  take,  and  that  they  were  free  to  take  short  breaks 
at  their  own  discretion. 

For  both  the  adaptive  and  nonadaptive  versions,  the  computer  session  started  and 
concluded  with  the  knowledge  test,  consisting  of  22  questions,  which  covered  84%  of  the  ELOs. 

Following  the  pretest,  Soldiers  doing  the  adaptive  version  read  through  a  personalized  set 
of  instructional  materials,  which  were  selected  on  the  basis  of  errors  made  on  the  pretest.  Once 
they  completed  this  instruction,  they  completed  the  offensive  scenario,  and  then  the  defensive 
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scenario,  according  to  the  procedures  described  in  the  adaptive  scenario  section  above. 

Following  the  pretest,  Soldiers  doing  the  nonadaptive  version  read  through  all  the  instructional 
materials.  Once  they  completed  this,  they  completed  the  offensive  and  defensive  scenarios 
(described  in  the  adaptive  implementation  section  above).  All  Soldiers  concluded  the  session  by 
completing  the  posttest,  and  two  questionnaires,  one  on  system  usability,  and  one  on  their 
opinions  concerning  the  training. 

Results  and  Discussion 

Soldiers  completed  the  pretest  (without  feedback)  prior  to  training  and  an  identical 
posttest  after  training.  On  average,  test  scores  increased  from  pretest  (48.3%)  to  posttest  (61.7%). 
A  mixed  analysis  of  variance  using  these  test  scores  as  the  repeated  factor  and  condition  as  the 
between-group  factor  indicated  a  significant  effect  of  training  (pretest  vs.  posttest),  F(l,  22)  = 
18.06 ,p  <  .001,  with  an  effect  size  (partial  eta-squared)  of  0.49.  There  failed  to  be  a  significant 
two-way  interaction  between  test  time  and  condition,  as  would  be  expected  if  the  adaptive 
version  produced  better  learning  outcomes  than  the  nonadaptive  version.  Mean  pretest  and 
posttest  scores  were  46.3%  and  60.7%  respectively,  in  the  nonadaptive  condition,  and  were 
50.3%  and  62.6%  respectively,  in  the  adaptive  condition. 

Soldiers  in  both  conditions  spent,  on  average,  about  the  same  amount  of  time  to  complete 
their  training  sessions.  The  mean  was  94.0  minutes  for  the  nonadaptive  condition,  and  102.9 
minutes  for  the  adaptive  condition.  The  range  of  times  varied  widely,  however;  the  fastest 
Soldier  completed  the  session  in  about  41  minutes,  whereas  five  Soldiers  took  three  hours. 

Figure  8  shows  that  Soldiers  who  took  more  than  85  minutes  (the  median)  to  complete  the 
session  increased  more  from  pretest  to  posttest  than  those  who  took  less  than  85  minutes  to 
complete  the  session,  F(  1,  20)  =  1 1.24 ,p  <  .01.  There  was  no  interaction  of  time-to-complete 
with  condition,  however  (F  <  1). 

Relevance  of  the  training  to  the  Soldiers’  branch  and  duties  during  their  last  deployment 
also  affected  test  scores.  Those  for  whom  the  training  domain  was  more  relevant  scored  higher 
on  both  the  pretest  and  the  posttest  (5 1 .6%  and  68.2%,  respectively),  than  those  for  whom  the 
training  domain  was  relatively  less  relevant  (44.3%  and  52.5%,  respectively),  F(  \ ,  20)  =  1 1.34;  p 
<  .01,  effect  size  partial  eta-squared  =  0.36.  See  Appendix  C  for  assignment  of  Soldiers  to  these 
conditions.  The  effect  of  relevance  failed  to  interact  with  condition  (F  <  1). 

After  training,  Soldiers  responded  to  eight  questions  about  the  usability  of  the  training 
system.  A  detailed  summary  of  the  questions  and  responses  can  be  found  in  Appendix  D.  On  the 
whole,  responses  were  moderately  positive,  but  indicated  that  the  system  or  the  introductory 
instructions  could  be  improved.  For  example,  only  about  70%  of  Soldiers  agreed  or  strongly 
agreed  that  they  understood  the  “knobology”  and/or  felt  confident  they  could  explain  how  the 
system  worked  to  a  friend.  We  failed  to  detect  any  differences  in  responses  according  to 
condition. 
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Vertical  bars  denote  standard  errors  of  the  mean 


Figure  8.  Mean  percent  correct  on  the  pretest  and  posttest,  according  to  time  on  task.  The  left 
panel  shows  the  data  for  participants  who  spent  less  than  the  median  amount  of  time  on  task  and 
the  right  panel  shows  the  data  for  participants  who  spent  more  than  the  median  amount  of  time 
on  task. 

After  training,  Soldiers  also  responded  to  14  questions  concerning  their  opinions  about 
the  training  effectiveness  of  their  experience.  A  detailed  summary  of  the  questions  and  the  modal 
responses  can  be  found  in  Appendix  E.  Overall,  the  responses  were  more  favorable  in  the 
adaptive  condition  than  the  nonadaptive  condition.  The  modal  responses  for  eight  of  the  14 
questions  favored  the  adaptive  condition.  For  the  remaining  questions,  the  modal  responses  were 
the  same.  This  comparison  represents  a  significantly  more  favorable  appraisal  by  those  taking  the 
adaptive  version,  Wilcoxon  T  =  0,p  <  .05.  Examining  each  question  individually  using  Mann- 
Whitney  U  tests,  there  were  three  questions  for  which  the  adaptive  condition  responded 
significantly  more  positively  than  the  nonadaptive  condition.  These  questions  were 

•  Experiencing  actual  situations  that  might  arise  during  the  employment  of  an 
SET  AS.  Adjusted  z  =  2.39,  p  <  .05. 

•  The  scenario(s)  I  worked  on  were  realistic.  Adjusted  z  =  2.10  ,p<  .05. 

•  The  hints  available  during  the  scenarios  were  helpful.  Adjusted  z  =  3.01,/?<.01. 
[Note  that  both  conditions  had  the  same  hints  available]. 
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A  summary  of  the  Soldier’s  session  logs  are  presented  in  Appendix  F.  Inspection  of  the 
log  data  reveals  that  performance  on  the  second  (Defense)  scenario  was  somewhat  worse  than 
the  Offense  scenario,  /  =  1 0.08,  p  <  .01.  Only  two  Soldiers  failed  to  pass  the  Offense  scenario, 
while  12  failed  to  pass  the  Defense  scenario.  Six  of  these  were  in  the  adaptive  condition,  where 
Soldiers  were  allowed  three  attempts  at  the  scenario.  It  may  be  that  the  Defense  scenario  was 
more  difficult  than  the  Offense  scenario;  or  it  may  be  that  Soldiers  were  simply  growing  tired  by 
this  time,  and  rushing  to  finish.  The  time  spent  on  the  Defense  scenario  (first  attempt  for  the 
adaptive  condition)  was  significantly  lower  (mean  7.4  minutes)  than  on  the  Offense  scenario 
(mean  15.5  minutes),  F(\ ,  22)  =  17.3;/?  <  .001.  Soldiers  also  took  significantly  less  time  to 
complete  the  posttest  than  the  pretest,  spending  on  average  12.0  minutes  on  the  pretest,  but  only 
7.1  min  on  the  posttest,  F(  1.22)  =20.66;  p  <  .001. This  could  be  because  Soldiers  found  the 
posttest  easier  than  the  pretest  (having  gone  through  the  training);  but,  it  could  also  could  be 
because  Soldiers  just  wanted  to  finish  the  session  as  quickly  as  possible,  and  so  deliberated  less 
on  each  question.  The  possibility  that  Soldiers  tended  to  rush  through  the  latter  part  of  the 
session  (including  the  posttest)  may  have  undermined  our  ability  to  evaluate  the  relative 
effectiveness  of  the  two  training  versions,  by  adding  substantial  noise  to  the  posttest  data. 


Conclusions  and  Lessons  Learned 

Soldier’s  test  scores  increased  significantly  from  pretest  to  posttest;  however,  the  results 
failed  to  demonstrate  significantly  greater  learning  from  the  adaptive  vs.  the  nonadaptive  training 
prototype.  As  with  all  failures  to  find  an  effect,  we  should  not  necessarily  conclude  there  is  no 
difference  in  the  effectiveness  of  the  prototypes,  because  there  are  multiple  possible  reasons  for  a 
failure  to  find  an  effect.  It  may  be  that  our  pretest  and  posttest  were  insufficiently  sensitive  to 
detect  differences  in  amounts  of  learning.  We  shortened  our  test  to  22  questions  because  pilot 
testing  indicated  that  Soldiers  could  take  more  than  an  hour  to  complete  the  longer  (42-item)  test. 
However,  a  longer  test  with  multiple  items  related  to  each  LO  would  have  provided  a  more 
sensitive  measure,  less  prone  to  noisy  data  resulting  from  guessing  or  haste. 

Soldiers  varied  widely  in  the  amount  of  time  they  took  on  each  phase  of  the  training.  In 
general,  Soldiers  who  took  more  time  tended  to  have  larger  increases  between  pretest  and 
posttest  scores  (Figure  8).  Amount  of  time  spent  on  each  succeeding  phase  tended  to  decrease. 
Soldiers,  on  average,  spent  about  twice  as  much  time  on  the  pretest  than  the  posttest.  This  likely 
reflected  combination  of  decreasing  interest  over  time  (resulting  in  a  rush  to  complete)  and  an 
increase  in  knowledge  (resulting  in  the  need  for  less  deliberation  over  each  question).  Perhaps  it 
was  unrealistic  to  conduct  the  amount  of  training  we  did  in  a  single  session. 

Even  if  we  accept  the  conclusion  that  the  adaptive  and  nonadaptive  conditions  failed  to 
affect  learning  differently,  that  conclusion  must  be  specific  to  the  particular  implementation  of 
the  training  software  tested  here.  The  present  results  do  not  rule  out  the  possibility  that  different 
parameters  or  different  methods  of  adaptation  might  be  more  effective.  In  the  present 
implementation,  we  gave  Soldiers  in  the  adaptive  condition  up  to  three  tries  at  each  scenario. 
Nevertheless,  five  Soldiers  in  this  condition  did  not  successfully  complete  one  or  both  scenarios. 
Consequently,  our  intention  to  enforce  a  mastery  criterion  was  not  actualized  in  their  cases. 
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Requiring  mastery  should  improve  posttest  scores  (Durlach  &  Ray,  2011),  so  increasing  this 
limit  might  produce  better  learning  outcomes. 

Another  aspect  of  our  adaptive  implementation  that  could  be  improved  was  the  method 
of  remediation.  We  used  materials  taken  from  the  upfront  instruction  to  provide  remediation. 
These  consisted  of  excerpts  from  handbooks  and  other  doctrinal  sources,  combined  together;  but 
these  materials  were  not  specifically  designed  for  instruction  or  the  particular  scenario  situations 
that  would  instigate  the  remediation.  A  review  of  the  learning  sciences  literature  (Durlach  & 

Ray,  2011)  suggests  that  our  adaptive  remediation  strategy  could  have  been  more  effective  if  we 
had  used  error-sensitive  remediation.  Error-sensitive  remediation  would  address  not  only  that  a 
student  erred,  but  also  the  way  in  which  they  erred.  Instead  of  using  abstract  conceptual  materials 
for  remediation  during  scenario-based  decision  making,  error-sensitive  remediation  would 
address  the  error  made  with  direct  reference  to  the  scenario  context.  Not  using  this  strategy  likely 
accounts  for  why  five  Soldiers  could  not  complete  the  scenarios  successfully  with  the 
remediation  provided. 

Reviewing  the  research  literature,  Durlach  and  Ray  (2011)  did  identify  some  experiments 
in  which  learning  outcomes  were  superior  for  an  adaptive  system  vs.  a  parallel  nonadaptive 
system  (e.g.,  Perrin,  Dargue,  &  Banks,  2003);  but,  they  also  discovered  several  experiments,  like 
this  one,  which  failed  to  find  any  outcome  differences.  They  concluded  that  there  can  be  learning 
benefits  from  adaptive  approaches  over  nonadaptive  approaches;  but,  that  the  benefits  are  not 
guaranteed.  Designing  adaptive  training  technology  may  need  to  be  an  iterative  process,  where 
cycles  of  development,  testing,  and  refinement  may  be  required  to  ensure  that  the  potential 
benefits  are  realized.  Current  heuristic  guidance  on  how  to  implement  adaptive  strategies  in 
software  are  not  specific  enough  to  ensure  that  the  first  version  of  any  particular  implementation 
will  be  beneficial  (compared  with  a  nonadaptive  implementation).  Systematic  documentation  of 
the  iterative  process  and  comparison  of  the  effectiveness  of  different  implementations  is  required 
to  transform  vague  heuristic  guidelines  into  more  specific  implementation  recommendations. 
When  engineering  a  system,  it  is  tempting  (and  common)  to  make  multiple  refinements  at  one 
time,  in  order  to  get  to  the  final  version  more  quickly.  Unfortunately,  this  works  against 
developing  an  understanding  concerning  the  relative  contributions  of  the  various  changes  to 
system  effectiveness,  and  in  turn  undennines  development  of  a  systematic  understanding  that 
will  take  us  beyond  mere  design  heuristics. 

Despite  our  failure  to  find  learning  differences,  Soldiers’  provided  more  positive 
appraisals  of  the  training  effectiveness  of  the  adaptive  version  over  the  nonadaptive  version.  The 
current  adaptive  version  may  provide  the  basis  for  the  development  of  improved  training  on 
SUAS  employment  at  company  and  below. 
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Appendix  A 


Task  Analysis  Sources 


Main  sources 

•  Leader’s  Guide  to  A2C2  at  Brigade  and  Below.  No.  05-25,  Center  for  Army 
Lessons  Learned  (CALL),  June  05. 

•  FM  3-52.  Army  Airspace  Command  and  Control  in  a  Combat  Zone  (FM  100- 

103),  Field  Manual.  Headquarters,  Department  of  the  Army,  Washington,  DC,  1 
August  2002. 

•  FMI  3-04.155.  Army  Unmanned  Aircraft  System  Operation.  Field  Manual 
Interim  Headquarters  No.  3-04.155  Department  of  the  Army.  Washington,  DC,  4 
April  2006  Expires  4  April  2008. 

•  Joint  Concept  of  Operations  for  Unmanned  Aircraft  Systems.  Creech  Air  Force 
Base,  Indian  Springs,  NV.  First  Edition,  March  2007. 

Other  Sources 

•  FM  3-04.1 1 1.  Aviation  Brigades.  21  August  2003. 

•  FM  3-04.126.  Attack  Reconnaissance  Battalions  (Draft).  6  June  2004. 

•  FM  3-06.  Urban  Operations.  1  June  2003. 

•  FM  3-20.95.  Cavalry  Operations.  (Final  Draft)  2004. 

•  FM  3-21.21.  Risk  Management  and  Fratricide  Avoidance.  31  July  2003. 

•  FM  3-90.  Tactics.  4  July  2001. 

•  FM  3-90.6.  The  Brigade  Unit  of  Action.  (Draft  TBD).  15  March  2005. 

•  FM  3-90.XX.  UA  Fires  and  Effects. 

•  FM  3 -90. XX.  UA  Maneuver  Operations. 

•  FM  5-0.  Army  Planning  and  Orders  Production.  20  January  2005. 

•  FM  7-15.  Army  Universal  Task  List  (AUTL),  Anny  Tactical  Task.  (ART)  8.1. 
Conduct  Offensive  Operations.  31  August  2003 

•  FMI  3-04.155,  Army  Unmanned  Aircraft  Systems  Operations,  April  2006. 

•  FMI  3-09.42.  HBCT  Fires  and  Effects  Operations.  15  April  2005. 

•  FMI  5-0.1.  The  Operations  Process.  March  2006 

JP  2-01.1.  Tactics,  Techniques  and  Procedures  for  Intelligence  Support  to  Targeting. 
9  January  2003. 

•  JP  3-09.3.  Joint  Tactics,  Techniques  and  Procedures  for  Close  Air  Support  (CAS).  3 
September  2003. 
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•  TRADOC  Pam  525-3-90.  The  United  States  Army  Future  Combat  Force  Operational 
and  Organizational  Plan  for  the  Future  Combat  Systems  Brigade  Combat  Team. 
Change  3.16  December  2005. 

•  FM  3-52  (FM  100-103)  .ARMY  AIRSPACE  COMMAND  AND  CONTROL  IN  A 
COMBAT  ZONE. 

•  JP  3-52  JOINT  DOCTRINE  FOR  AIRSPACE  CONTROL  IN  THE  COMBAT  ZONE. 

•  JP  3-55.1,  Joint  Unmanned  Aerial  Vehicle  Regulation,  27  August  1993. 

•  AR  95-23.  Unmanned  Aerial  Vehicle  Flight  Regulations.  14  May  2004. 

•  ARTEP  34-1 17-30-MTP.  Mission  Training  Plan  for  the  Surveillance  Troop  of  the 

•  Stryker  Brigade  Combat  Team.  28  June  2005. 

•  ARTEP  34-387-10-DRILL.  Crew  Drills  for  the  Shadow  200  Tactical  Unmanned 

•  Vehicle  (TUAV).  20  November  2005. 

•  ARTEP  34-387-30-MTP.  Mission  Training  Plan  for  the  Direct  Support  Military 

•  Intelligence  Company  (Digitized).  30  September  2004. 

•  ARTEP  34-414-10-DRILL.  Crew  Drills  for  the  Hunter  Tactical  Unmanned  Aerial 

•  Vehicle  (TUAV).  21  February  2006. 

•  ARTEP  34-414-30-MTP.  Mission  Training  Plan  for  the  Aerial  Reconnaissance 

•  Company  (Aerial  Exploitation)  (Corps).  2  December  2002. 

Additional  Resources  were  consulted  from  AKO  and  Other  Internet  sources.  The  following  were 
most  useful: 

•  NFTF  -  MAJ  Chris  Barra.  Airspace  Command  and  Control  at  the  Brigade  Level.  June 
2005 

•  SAMS  Monograph.  Rise  of  the  unmanned  aerial  vehicle  and  its  effect  on  manned  tactical 
aviation.  MAJ  James  Meger.  US  Army  Command  and  General  Staff  College,  1  Reynolds 
Ave.,Fort  Leavenworth,  KS.  June  2006. 

Most  joint  publications  can  be  found  at  http://www.dtic.mil/doctrine/index.html. 

Most  Army  publications  are  available  at 
https://akocomm.us.army.mil/usapa/doctrine/index.html. 
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Appendix  B 


Relation  Among  Learning  Objectives  and  Source  Data 


LO  ID 

Objective 

Source  Data 

1_SUASROLE 

Understand  the  role  of  the  SUAS  in  the  overall  tactical 
plan/operation. 

5-25  CALL  HB;  Appendix  B; 
FM  3-04.15  Ch  II  or  IV. 

SUAS  Tactical 

Basics 

Objective  of  Didactic  Training:  Understand  the 

SUAS  is  used  for  RSTA,  Security,  and  Battle  Damage 
Assessment.  At  the  SUAS  it  is  not  an  attack  platform. 
Understand  that  the  performance  of  these  missions 
RSTA,  Security,  and  BDA  requires  airspace 
coordination  and  ground  coordination  to  support  the 
overall  tactical  plan.  Be  exposed  to  the  doctrinally 
correct  definitions  and  a  visual  representation  of  a 

SUAS  in  the  appropriate  context  (e.g.  R&S  mission  with 
airspace  coordination  implications  displayed). 

1A_SUAS-OPN 

Attack,  Defense, 
Security,  Civil 
Operations 

Understand  the  role  of  the  SUAS  in  the  major  Army 
operations  (Attack,  Defense,  Security,  Civil 
Operations). 

FM  3-90  Chapter  2  (pp.  2-1 
thru  2-39;  FM  3-04.15,  Ch  1 
(Man-portable  SUAS  focus); 
Summarize  FM  3-04.15  para 

6.  a,  b.c.d.  (p.  I-7&  1-8)  and 

FM  3-90  p.  2-1  and  p.  2-2 
Include  Figure  2-1.  Capture 
content  of  Par  2-1  thru  2-3 
(p.  2-1  thru  2-2)  See  FCS  IMI 
Lsn  3,  frame  73-75. 
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1  B_SUAS-R&S 

R&S 

Know  the  essential  elements  and  concepts  of 

Reconnaissance  and  Surveillance  (R&S)  SAUS 
missions. 

FM  3-90,  Ch.  12  focused  on 
Company  Level  ,  FM  3-20.98 
Chapter  3  &  Chap  3:Sec  II, 

FM  3-04.15  Ch  IV 
(Specificsdntro  -  Recon- 
Definition  in  FM3-04.15  in 
Glossary  p.  9  followed  by  Fig 
IV-1  then  section  on  Route 
Recon  -para  2.a.(1).  (a)  and 
(b),pp.  IV-1  thru  IV-2);  then 
Zone  Recon  para  2.b.  (1), 

(2),  (3)  p.  IV-3);  Area  Recon- 
para  2.c.(1),  (2), (3), (4),  p.  IV- 
4)  Surveillance  -  Definitiion, 
p.9  of  Glossary,  FM3-04.14 
put  copy  of  DDForm  1975  to 
illustrate,  and  then  Summary 
of  para  3.a.,b.,c.,d.,e.  (make 
sure  you  write  out  all  terms 
like  SPINS  (Special 
Instructions)  by  going  to 
Glossary),  p.  IV-4  thru  IV-5. 

1C_SUAS-TA 

T&A &  BDA 

Know  the  essential  elements  and  concepts  of  Target 
Acquistion  (TA)  and  Battle  Damage  Assessment 
(BDA)  SAUS  missions. 

FM  3-04.15,  Chapter  IV 
(Specifics-  Intro  Definition  in 
Gloassary,  FM3-04.15, 
p. Glossary-9,  Summarize 
para.  4.  a.  Al.  (3)only  then  4. 
b.  with  emphasis  on  (2)&(5) 
with  emphasis  on  Fire 

Support  Coordination 
Measures  &  MRR. 
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1  D_SUAS-SEC 

Security 

Know  the  essential  elements  and  concepts  of  Security 
in  SAUS  missions.  Focus  on  Screen  and  Force 

Protection. 

FM  3-90,  Ch.  12  focused  on 
Company  Level  See  also  FM 
3-90.98.  See  also  FM3.04.15 
Ch  IV. (Specifics:  Definition  in 
FM3-04.15  in  Glossary  p.  9 
Define:  Security  mission  of 
Screen,  Guard,  and  Cover 
then  define  Force  Protection 
(FP)  see  FM  3-04.15  p.7  in 
Glossary  and  the  go  to  FMI3- 
04.155  and  summarize  par 
3-38  thru  3-41  (p.  3-10) 
omitting  technical  data  but 
focusing  on  mission  and 
objectives.  See  also  para  4- 
135  &  4-136  on  p.  4-25of 
FMI3-04.155  emphasizing 
the  situation  awareness 

SUAS  gives  the  Cdr  for  FP  a 
sensor  to  mitigate  risk 
against  certain  threats. 

Really  force  protection  is  like 
a  screen  but  in  a  "non 

combat"  mode.  FM3-04.155 
para  5-122  third  bullet  is  a 
good  summary  force 
protection. 

1 E_SUAS_BDA 

BDA 

Know  the  essential  elements  and  concepts  of  Battle 
Damage  Assessment  SAUS  missions. 

Know  FM  3-04.15  Par  6  a.  & 
b.(p. IV-1 4)  A  part  of  Combat 
Assessment  (See  FM  3- 
04.15  Glossary  5).  See  also 
FM3-04.155  par  B-41  (p.B-7) 
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and  para. 5-54  (p.5-13)  and 
CALL  HB  5-25  p.29  as  a 
primary  mission  of  SUAS. 

2_TACTC0NT 

Understand  common  company  and  battalion  tactical 
control  measures. 

FM  3-90  &  FM  3-52 

SUAS  Bn  &  Comp 
Tac  Control 

Measures 

Objective  of  this  Didactic  Training:  Be  able  to  apply 
the  definitions  and  graphical  symbols  associated  with 
basic  offense  and  defensive  Army  battalion  operations 
and  understand  their  basic  implications.  The  following 
are  not  all  inclusive  of  the  basic  tactical  control 
measures  but  are  essential  to  this  given  training.  This 
section  should  be  designed  to  be  expanded  to 
accomodate  other  control  measures  like  Limit  of 

Advance  (LOA)  FM  3-90  (p.3-7).  Others  like  Line  of 
Departure  (LD)  are  assumed  to  be  understood  by  the 
student  as  they  are  basic  Army  terminology. 

2A_TACT_NAI 

NAI 

Understand  the  importance  of  positive  control  during 
observation  of  Named  Area  of  Interest  (NAI). 

FM  3-90,  p.  2-33;  FM  3-52 
sec  5-43  thru  5-47  p.  5-1 1 . 

2B_TACT-AA 

Axis  of  Advance 

Understand  the  concepts  of  Axis  of  Advance 

FM  3-90,  p.  3-5;  FM  3-52  sec 
4-16  (p.4-7) 

2C_TACT-BP 

Battle  Position 

Understand  the  concepts  of  Battle  Position 

FM  3-90,  p.  8-7;  FM  3-52  sec 
4-18,  p.4-7. 

2D_TACT-EA 

Engagement  Area 

Understand  the  concepts  of  Engagement  Area 

FM  3-90,  p.  2-21;  FM  3-52 
sec  4-19,  p.  4-7. 

2E_TACT-ABF 

Attack  by  Fire 
Position 

Understand  the  concepts  of  Attack  by  Fire  Position 

FM  3-90,  p.3-5;  FM  3-52  sec 
4-21,  p.4-8. 

2F_TACT-0P 

Observation  Post 

Understand  the  concepts  of  Observation  Post 

FM  3-90,  p.  12-5  thru  12-10; 
FM  3-52  sec  4-22;  p.4-8. 
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3_PR0-SEPC 

Understand  and  know  the  three  types  of  separation 
used  in  manned  and  unmanned  aircraft.  (5-25  CALL, 
p.4;  FM  3-52,  p.  4-12) 

5-25  CALL,  p.4;  FM  3-52,  p. 
4-12.  See  FM  3.04.15 

Chapter  II.  See  FM3-04.15 
Chap  II,  par  5. a,  e.f,  g.  h,  i., 
j.;  FM3-52,  p.4-20  thru  4-11 
para.  4-32  and  4-34.  Maybe 
use  Figure  4-9  with  SUAS 
added. 

Airspace  Methods 
of  Separation  - 
Procedural  Controls 
-  Separation 

Objective  of  Didactic  Training:  This  should/could  be 
a  adaptation  of  the  FCS  IMI  Lsn  2  ELO  4 

Identify  Employment  Considerations  for  UAS  Airspace 
Deconfliction  Requirements 

This  ELO  focuses  on  information  for  company/platoon 
leadership  on  conducting  airspace  deconfliction  for  the 
Class  1  UAS  (Blk  0),  to  include  the  following: 

--  Operations  in  the  vicinity  of  other  UAS,  and  manned 
aircraft 

--  Operations  in  the  vicinity  of  friendly  fires 

3A_PR0-LAT 

Lateral 

Understand  and  know  Lateral  separation 

5-25  CALL  HB,  p.4;  FM3- 
04.155  B-16,  p.B-3 

3B_PR0-TIM 

Time 

Understand  and  know  Time  separation 

5-25  CALL  HB,  p.4;  FM3- 
04.155  B-17,  p.B-3 

3C_PR0-VET 

Vertical 

Understand  and  know  Vertical  separation 

5-25  CALL  HB,  p.4;  FM3- 
04.155  B-18,  p.B-3 

4_MIS-REQ 

Understand  the  two  types  of  airspace  mission 
requests  and  three  types  of  missions 

5-25  CALL  HB  Ch  2  (pp.10- 
13) 
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Mission  Types  and 
Mission  Requests 

Objective  of  Didactic  Training:  Know  the  definitions  of 
mission  requests  and  the  general  format  of  a  immediate 
SUAS  mission  request.  See  the  RAVEN  PO1 11-2  LSA 

4  Airspace  Management  pdf  pp.  1 1-18  for  essential 
information  to  capture.  See  also  Lsn  2  FCS  Planning 
Consideration  IMI  frame  84-99  (see  frame  93  for 
immediate  mission  request  format. 

Summary  of  SUAS  mission  request  types  : 

Planned  --  Done  during  Planning  phase  (more  than  72 
hrs  prior  to  LD)  -  PLANNED  REQUEST 

Immediate  --  done  during  the  Prepare  phase  (less  than 

72  hrs  but  before  LD)  -  IMMEDIATE  REQUEST 

Dynamic  --  done  during  Execute  phase  (after  LD)  - 
IMMEDIATE  REQUEST  per  UNIT  SOP 

All  types  can  be  planned  for  and  trained  for  as  they  are 
variants  of  the  same  procedure  as  are  OPLANs, 
OPORDs,  and  FRAGOs. 

**  See  John  Sanders  for  this 

TLO  as  he  has  an  excellent 
simplification.  The  FCS 
frames  are  also  well  done  for 
didactic  training. 

4A_MIS-PLAN 

Planned 

Understand  the  considerations  and  coordination 
required  for  a  planned  SUAS  mission  request,  (more 
than  72  hrs  prior  to  mission  start) 

5-25  CALL  HB  Ch  2  (pp.12); 
p.25-26; 

4B_MIS-IMMD 

Immediate 

Understand  the  considerations  and  coordination 
required  for  an  immediate  SUAS  mission  request. 

(done  less  than  72  hrs  prior  to  mission  start  but  not  after 
mission  start) 

5-25  CALL  HB  Ch  2  (pp.12- 
13);  p.27 

4C_MIS-DYNAMIC 

Dynamic 

Understand  the  considerations  and  coordination 
required  for  a  dynamic  re-tasking  SUAS  mission 
request. 

FM  3-04.15  p.  11-5  thru  11-7.  Dynamic  flight  occurs 
anytime  a  platform  deviates  from  its  planned  mission — 

FM  3-04.15  pp.  11-5  thru  11-8. 
See  IMI  Lsn  2  pp.  62-82. 
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a  common  occurrence  during  combat  operations  (done 
during  a  mission  execution  (after  Line  of  Departure). 

Use  immediate  SUAS  mission  request  format  modified 
and  adapted  if  required. 

5_DEC0N-0 

Understand  the  importance  of  deconfliction  in 
airspace  coordination 

5-25  CALL  HB,  p.4  &  Ch.  4 
(14-20),  FM  3-04.15  Ch  1  & 

III. 

SUAS  and 

Deconfliction 

Objective  of  this  Didactic  Training:  This  section  is 
the  practical  applications  of  airspace  management. 

These  are  the  major  types  of  aircraft  (manned  and 
unmanned/Army)  and  fires  that  the  SUAS  must  not 
adversely  impact  but  complement.  The  emphasis 
whould  be  show  visual  examples  of  the  need  to 
deconflict  via  diagrams  from  doctrinal  literature  or  POI. 
See  and  Use  FCS  IMI  Lsn  2  ELO  4  pp.  83-94.  Include 
some  information  of  deconflicting  communications  for 
UAVs,  Manned  Aircraft,  and  direct/indirect  fires  (see  FM 
3-04.15  pp.  111-1  thru  111-3  (Section  1.  para  b  &  c.) 

5A_DEC0N-UM 

Other  SUAS/UAVs 

Understand  the  importance  and  coordination  necessary 

to  deconflict  other  UAV  operations. 

5-25  CALL  HB,  p.4  &  Ch.  4 
(14-20);FM  3-04.15  pp.  111-1 
thru  111-3  (Section  1.  para  b  & 

c 

5B_DEC0N-MA 

Manned  Aircraft 

Understand  the  importance  and  coordination  necessary 

to  deconflict  other  Manned  Aircraft  operations. 

5-25  CALL  HB,  p.4  &  Ch.  4 
(14-20);FM  3-04.15  pp.  111-1 
thru  111-3  (Section  1.  para  b  & 

c 

5C_DEC0N-IND 

Direct  and  Indirect 

Fires 

Understand  the  importance  and  coordination  necessary 
to  deconflict  with  direct  and  indirect  fires. 

5-25  CALL  HB,  p.4  &  Ch.  4 
(14-20);FM  3-04.15  pp.  111-1 
thru  111-3  (Section  1.  para  b  & 

c 

B-7 


6_SA-0PN 

Understand  the  needs  and  necessity  of  coordination 
with  other  units  affected  by  SUAS  operations  to 
support  situational  awareness.  See  HB  CALL  5-25 
para  1,  p.  11. 

SUAS  Staff 

Coordination  and 

Situation 

Awareness 

Considerations 

Objective  of  Didactic  Training:  The  SUAS 
commander/operator  must  coordinate  below,  above, 
and  to  their  left  and  right  to  ensure  the  SUAS  flight  and 
operations  are  synchronized.  This  section  could  be  an 
adaptation  of  FCS  IMI  Lsn  1  pg  53-71.  Also  it  could  be 
summary  of  Figure  111-1  and  111-2  UAS  C2  in  FM  3- 
04.15.  FM  3-04.155  Appendix  B  Figure  B-6,  pp.  B-14. 

6A_SA-INF0 

Other  Units  (Above, 
below,  adjointing) 

Understand  the  importance  of  sharing  information 
between  units  and  higher  echelons  as  part  of  mission 
planning.  (5-25  CALL,  p.7-11) 

5-25  CALL  HB,  p.7-11,  p.  21 

6B_SA-0BS 

Observation  Posts 
concepts  for 
Situational 
Awareness  (SA) 

Know  the  essential  elements  and  concepts  of 
Observation  to  support  situational  awareness  SAUS 
missions. (5-25-, CALL, p.1 1 ) 

5-25  CALL  HB,  p.ll.p.  21. 
See  also  FM  3-90,  p.  12-7 
and  others.  Summarize  FM 
3-04.15  Ch  IV,  section  3,  pp. 
IV-4thru  IV-5  on  Surveillance 
with  a  focus  on  Brigade  and 
below  applications. 

6C_SA-BDA 

Battle  Damage 
Assessment 

Know  the  essential  elements  and  concepts  of  Battle 
Damage  Assessment  dissemination  during  SAUS 
missions.  (5-25  CALL,  p.  1 1 )  This  is  the  same  as  1 E. 

Use  the  same  didactic  training. 

5-25  CALL  HB,  p.1 1, 
Summarize  Sec  6  of  Ch  IV  of 
FM  3-04.15  (pp.  IV-14) 

7_RISK-ENT 

Understand  the  risks  and  likelihood  in  the  Airspace 
Environment.  (FM3-04.155,  Appendix  A  )  *ln  my 
research  for  this  objective  there  are  usually  'operational 
considerations'  for  employment  of  SUAS  e.g.  wind 

FM3-04.155,  Appendix  A&B, 
FM  3-04.15  Ch  II, p.  11-16  Sec 

8  (Environmental 
Considerations  in  Mission 
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speed.  1  have  attempted  to  document  the  likely  risks 
which  a  Skill  Level  2  up  to  Company  Grade  officer 
should  be  aware. 

Planning),  5-25  CALL  HB  Ch 

1. 

SUAS  Airspace 
Environment  and 

Threats/Risks 

Objective  of  Didactic  Training:  The  SUAS 
commander/operator  should  understand  the  potential 
"obstacles"  in  the  airspace  -  natural,  man-made,  and 
account  for  their  presence  in  planning  and  operations. 
This  section  encompasses  the  majority  of  ELO  5  from 
FCS  IMI  Lsn  2  pg  100-107  and  FM  3-04.155  and  FM  3- 
04.15  extracts.  Specifics  follow. 

Selected  sections  from 
Appendix  A  and  B  of  FM  3- 
04.155  and  FM  3-04.15 

sections. 

7A_RISK-PL 

PowerLines  & 

Commo 

Interference 

Understand  the  risks  and  likelihood  of  Overhead 

Power  Lines  and  Communications  interference  in 

the  Airspace  Environment. 

FM3-04. 155, Table  A-1:UAS 
mission  planning  Checklist- 
Communication  Plan  section 
;  FM3-04.15  Ch  III. 

7B+C_RISK-T0W 

Man-made 

Structures 

Understand  the  risks  and  likelihood  of  Man-made 
structures  (Buildings,  Towers,  etc.)  in  the  Airspace 
Environment. 

FM3-04.155,  Chapter  4, 
section  4-27-29,  p.4-4 

Terrain&  Weather  section 

7D_RISK-LAF 

Low  altitude  civilian 

aircraft 

Understand  the  risks  and  likelihood  of  Low  altitude 
civilian  aircraft  in  the  Airspace  Environment. 

FM  3-04.155  Sec  II, 

Appendix  B  B-42-B-44  (p.B- 
7) 

7E_RISK-ART 

Close  Combat 

Attack  Aviation 
(Apache  Hel  -  AH- 
64) 

Understand  the  risks  and  likelihood  of  Artillery 
projectiles  and  effects  in  the  Airspace  Environment. 

FM3-04. 155, Section  IV  p.  5- 
24  thru  5-25  par  5-93thru  5- 
97;  FM3-52,  Glossary  p.7 
(CFL  ,RFL,  Definition) 

7G_RISK-TER 

Terrain  and  Line  of 
Sight 

Understand  the  risks  and  likelihood  of  Intervening 
Terrain  and  Line  of  Sight  implications  in  the  Airspace 
Environment.  (FM3-04.155,  Section  4-140  Terrain,  p.  4- 
25) 

(FM3-04.155,  Section  4-140 
Terrain,  p.  4-25) 

7G_RISK-TER 

Positive  Control 

and  Data  Links 

Understand  the  importance  of  planning,  preparing,  and 
executing  to  maintain  positive  control  of  SUAS  assets. 

CALL  HB  5-25,  pp  1-3; 

B-9 


7I_RISK-ABN 

Close  Combat 

Attack  Aviation 
(Apache  Hel  -  AH- 
64) 

Understand  the  risks  and  likelihood  of  Army  Aviation 
(Close  Combat  Attack)  operations  in  the  Airspace 
Environment. 

FM3-04.15  CCA  Handover 
Vignette  until  par  g.  pp.  IV- 
20-IV-21  then  through  IV-22 
where  CCA  is  topic. 

7J_RISK-ADA 

Enemy  Air  Defense 

Understand  the  risks  and  likelihood  of  Enemy  air 
defenses  (primarily  small  arms  fire)  in  the  Airspace 
Environment. 

FM3-04.155,  Chapter  4, 
section  4-34  (p.  4-5)  and  4- 
43  (4-7)  and  5-44  (p.5-11) 

7K_RISK-HAF 

CAS 

Understand  the  risks  and  likelihood  of  Close  Airs 
Support  (CAS)  aircraft  in  the  Airspace  Environment. 

FM3-04.15  pp.  IV-9  thru  IV- 
10  (Begin  para.b.  Close  Air 
Support  through  para  b.(5)). 

7  L  M  N  F  RISK- 

WEA 

Weather 

Understand  the  risks  and  likelihood  of  Weather 
(Precipitation,  winds,  tempature,  fog,  icing,  etc.)  in 

the  Airspace  Environment.  (FM3-04.155,  Chapter  4, 
section  4-31-34) 

FM  3-04.155  Chapter  4, 
section  4-31-34. 

8_C0NT-PR0 

Understand  the  procedural  control  measures  used  for 
battalion  and  below  operations  as  they  apply  to  SUAS 
operations  (FM  3-52,  1-6,  Ch  4) 

FM  3-52,  p.1-6,  Ch  4; 
5-25CALL  HB  5-25  pp  2-3 
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SUAS  Opns  & 
Procedural  Controls 
Measures 


Objective  of  Didactic  Training:  Objective  is  for  the 

Soldier  to  have  a  knowledge  of  the  basic  fire  support 
and  airspace  procedural  measures  at  a  conceptual 
level  of  understanding.  Know  the  defintion  and  several 
possible  uses  of  the  control  measure. 

Soldier  needs  to  know  conceptually  the  difference 
between  Formal  Airspace  Coordination  Area  and 
Informal  Airspace  Coordination  area  (FM  3-52) 

Soldier  needs  to  know  common  airspace  control 
measures  of  FM  3-52  (para  4-4,  p.4-2,  FM  3-52) 

--  Coordinating  Altitude  (para.4-5,  p.4-2, FM  3-52) 

--  Minimum  Risk  Route  (para  4-7,  p.4-3,  FM  3-52) 

--  Restricted  Operation  (para  4-8,  p.4-4,  FM  3-52) 

--  Army  Airspace  Control  Measures  SOPs  (  para  4-14 
thru  4-22),  p.  4-5-4-7,  FM  3-52). ..Air  Corridor,  Air 
Control  Point,  Communication  Checkpoint  (ACP) 

Soldier  need  to  know  major  Fire  Support  Coordination 
Measures  used  at  Brigade/Battalion/Company/Platoon 

--  Permissive  FSCMs  par  4-28  thru  par  4-31 ,  p  4-9 
thru  4-10,  FM  3-52. 

—  Restrictive  FSCMs  par  4-32  thru  4-27,  p.4-10  thru 
4-12,  FM  3-52 

Soldier  needs  to  know  uses  of  Army  Air  Control  Points 
(FM  3-52) 

Soldier  needs  to  know  uses  of  Air  Corridors,  Air 
Corridors,  ROZ/ROAs,  and  Minimum  Risk  Corridors. 


Summary  information  for 
incorporation  throughout  at 
FM  3-05.15  para 
(5), Airspace  Control 
Measures  p.  IV-10. 
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(FM  3-52) 

8A_C0NT-FAC 

Formal  Airspace 
Coord  Area 

Know  the  purpose  of  Formal  Airspace  Coordination 
Area.  (FM  3-52) 

FM  3-52  para  4-33,  p.  4-10  & 
4-11  &  FM  3-04.155  par  4- 
95  thru  4-103  (p.4-1 7thru  4- 
18). 

8B_C0NT-IAC 

Informal  Airspace 
Coord  Area 

Know  the  purpose  of  Informal  Airspace  Coordination 
Area.  (FM  3-52) 

FM  3-52, para.  4-34,  p.  4-11 
para  4-33,  p.  4-10  &  4-1 1  & 
FM  3-04.155  par  4-95  thru  4- 
103  (p.4-1 7thru  4-18). 

8C_C0NT-RFL+ 
8D_C0NT-RFA  + 

8D C0NT-NFA 

RFL  &  RFA  &  NFA 

Know  the  military  graphical  control  for  the  Restrictive 

Fire  [Control]  Line  (RFL)  and  Restrictive  Fire  Area 
(RFA)and  understand  its  purpose.  Know  the  military 

FM  3-52,  para  4-35  thru  4- 
37, (p.4-1 1  thru  4-12); 
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graphical  control  for  the  No  Fire  Area  (NFA)  and 
understand  its  purpose.  Know  the  military  graphical 
control  for  the  Restrictive  Fire  Area  (RFA)  and 
understand  its  purpose. (FM  3-52) 

CRS 

Know  the  defintion  of  a  Common  Reference  System 

(CRS). 

FM  3-52,  para  4-40,  p.  4-12 
and  Glossary  p.  6. 

8E_C0NT-ACP  & 
8F_C0NT-RCP  & 
8G_C0NT-CCP 

ACP 

Know  the  military  graphical  control  symbol  for  Air 

Control  Point  (ACP)  and  when  it  is  appropriate  to  use. 
Know  that  Air  Control  Point  (ACP)  use  can  rapidly 
facilitate  rapid  restructuring  of  routes.  Know  that 

communications  checkpoints  can  be  ACPs(FM  3- 

52). 

FM  3-52,  p.  4-7.  See 

Glossary  pg.  1  for  definition. 
FM  3-52,  p.4-8,  See 

Glossary  pg  6  for 
communication  checkpoint. 

FM  3-52,  p.4-8,  See 

Glossary  pg  6  for 
communication  checkpoint. 
See  FCS  IMI  Lsn  2,  frames 
88-89.  See  also  FCS  IMI, 

Lsn  2,  frame  88-89. 

TBD 

TBD 

TBD 

TBD 

8H_C0NT-AC 

Air  Corridors 

Know  the  military  graphical  control  symbol  of  Air 
Corridors  and  when  it  is  appropriate  to  use.  (5-25 

CALL,  p.2,  FM  3-52,  p.4-6). 

5-25  CALL,  p.2;  FM  3-52,  par 
a  4-16,  p.4-6  and  Glossary  p. 

1;  also  called  flight  corridor 

8I_C0NT-CCA 

AGL 

Know  that  the  air  altitudes  in  an  air  corridor  cannot 
exceed  the  coordinating  altitude. 

FM  3-52,  p.  4-6 

8J_C0NT-CCA 

AGL 

Know  where  to  find  Coordinating  Altitude  procedural 
airspace  control  information  in  a 
Brigade/Battalion/Company  OPLAN/OPORD.  (FM  3-52, 
P-4-2) 

FM  3-52,  para.  4-5,  p.4-2 
and  Glossary  p.  7;  See  FM 
3-52  Figure  4-1  pon  p.  4-3. 
Emphasis  on  para  3-72, p.  3- 
15,  FM3-52. 

8K_C0NT-R0Z 

ROZ/ROA 

Know  the  military  graphical  symbols  for  Restricted 
Operating  Zones/Areas  (ROZ/ROA)  and  when  it  is 

FM  3-52,  para  4-8  or  para  4- 
9,  p.4-4 
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appropriate  to  use.  (FM  3-52,  p.4-4) 

8L_C0NT-MRC 

Minimum  Risk 

Corridor 

Know  the  definition  of  Minimum  Risk  Corridor 
procedural  airspace.  (FM  3-52,  p.4-3)' 

FM  3-52,  para  4-7  and  Fig  4- 
2,  p.4-3  Glossary  p.  15. 

9_RESP 

Know  roles  and  responsibilities  of  BCT  Airspace 
coordination  teams  and  Emergency  Procedures 

SUAS  BCT 

Airspace 

Coordination  Team 

Objective  of  Didactic  Training:  This  section  should 
essential  roles  and  tasks  of  ADAM/BAE, Bn  S3/Bn 
S3Air,and  Company/SUAS  team  for  Airspace 
coordination 

Below  is  doctrinal  summarizations: 

ADAM/BAE  -  Summarize  FM3-04.155  Appendix  B 
Section  B-23  thru  B-26,  p.B-4 

Bn  S3/Bn  S3Air  -  Summarize  FM3-04.155  Appendix  B 
Section  B-28  and  B-29  p.  B-5 

Company  Responsibilities  -  Summarize  SUAS  Mission 
Planning  Flow  in  CAL  HB  5-25,  pp.  7-9  with  emphasis 
on  company  part  of  this  process. 
Recovery/Destruction/Abort/Emergency  Procedures  - 
Summarize  Appendix  E,.  Sec  II-  Raven  Recovery,  FM 
3-05.155.  Summarize  par  E-22-24.  This  should  be 
generalized  for  all  SUAS. 

9A_RESP-BAE 

ADAM/BAE 

No  SUAS  flies  without  prior  airspace  coordination 
through  the  ADAM/BAE. 

CALL  HB  5-25,  pp  4-6; 

9B_RESP-BnS 

Bn  S3/BnS3Air 

Understand  Bn  S-3  and  Bn  S-3  Air  staff 
responsibilies  tom  manage  SUAS  airspace 
coordination.  “S3  Air”  within  S3  has  staff 
responsibility  for  ALL  SAUS  operations.  (5-25 

CALL,  p.  6) 

(CALL  HB  5-25,  p.  6  and 
FM3-04.155  par  4-59  and  4- 
60  on  p.  4-10. 
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9C_RESP-Tm 

Company/T  rp/T  earn 

Understand  Company/SUAS  team  responsibilities  in 
airspace  coordination.  Know  that  improper  coordination 
may  result  in  catastrophic  damage  to  aircraft  and 
Soldiers. 

CALL  HB  5-25,  pp  7  & 
appendix  A 

9D_DESTRY 

Emergency  Opns 

Understand  Destruction/Abort  and  Emergency 
Procedures  for  SUAS  flight  planning  and 
operations. 

para.  E-22  thru  E-24  of 
Section  ll,Appendex  E,  FM 
3-04.155,  ppE-6  &  for 
context  JUAS  CONOPS,  pp 

11-1 6thru  11-17  (3)  Mission 
Planning  para  a  &  b. 
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Categorization  of  Soldiers  according  to  relevance  of  the  training  domain  to  their  last  duty 
position  while  deployed. 


More  Relevant 

Branch 

Duty  Position 

Armor 

Platoon  Leader 

Armor 

Troop  XO 

Aviation 

Flight  Platoon  Leader 

Field  Artillery 

BN  targeting  NCO 

Field  Artillery 

Pit  Sgt 

Infantry 

Platoon 

Leader/ASST  S3 

Infantry 

SquadLeader 

Infantry 

Platoon  Leader 

Infantry 

SquadLeader 

Military  Intelligence 

SignitPlatoon  Leader 

Military  Police 

Battle  Captain 

Signal 

Platoon  Leader 

Signal 

Co  XO 

(not  given) 

Squad  Leader 

Less  Relevant 


Branch 

Duty  Position 

Aviation 

Aviation  Mechanic 

Aviation 

Shop  Tech 

Engineer 

PL 

Field  Artillery 

CBRN  NOIC 

Military  Police 

Human  Resources 

Sgt 

Ordnance 

Shop  foreman 

Ordnance 

(not  given) 

Signal 

PLT  SGT(E7) 

Transportation 

BG  Transportation 
NCO 

Transportation 

TruckDriver 
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Appendix  D 

System  Usability  Questionnaire 


For  each  question.  Soldiers  were  asked  to  rate  their  level  of  agreement  with  the  given  statement 
as  Strongly  Agree,  Agree,  Neutral,  Disagree,  Strongly  Disagree.  For  purposes  of  analysis  these 
answers  were  recoded  as  2,  1,  0,  -1,  and  -2,  respectively.  No  statistically  significant  relationships 
were  found  between  the  training  outcomes  and  the  opinions  provided. 

Percent  of  Soldier  Responses 


Strongly 

agree 

Agree 

Neutral 

Disagree 

Strongly 

disagree 

1 . 1  felt  I  understood  the  “knobology”  of 
the  svstem  (how  to  make  it  work),  during 
the  instructional  and  testing  phases. 

29.2 

41.7 

16.7 

12.5 

0 

2. 1  felt  I  understood  the  “knobology”  of 
the  training  system  (how  to  make  it  work), 
during  the  scenario  phases. 

20.8 

50.0 

16.7 

12.5 

0 

3. 1  was  aware  that  I  could  ask  for  hints 
during  the  scenario  phases. 

62.5 

25.0 

8.3 

4.2 

0 

4. 1  was  aware  that  there  were  reference 
materials  about  the  SUAS  system  that  I 
could  access  during  the  scenario  phases. 

29.2 

33.3 

20.8 

16.7 

0 

5. 1  was  aware  that  there  was  a  glossary 
that  I  could  reference  during  the  scenario 
phases. 

33.3 

20.8 

20.8 

20.8 

4.2 

6.  The  effort  needed  to  operate  the 
training  system  was  manageable. 

29.2 

66.7 

4.2 

0 

0 

7. 1  am  confident  that  I  could  explain  to  a 
friend  how  to  use  this  system. 

25.0 

45.8 

25.0 

0 

4.2 

8. The  training  system  was  clear  about 
what  it  thought  I  needed  to  leam. 

12.5 

70.8 

16.7 

0 

0 
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Appendix  E 

Training  Evaluation  Questionnaire 
Modal  Responses  for  each  condition  for  each  question 


Thinking  about  the  training  prototype  you  worked  with  today, 
please  rate  it  on  how  you  think  it  would  compare  in 
effectiveness  to  conventional  classroom  training  on  each  of 
the  following: 

ADAPTIVE 

NONADAPTIVE 

1 .  Providing  knowledge  background  on  employment  of 

SUAS 

Better 

Just  as  good 

2.  Getting  exposure  to  the  kind  of  decisions  leaders  will  need 
to  make  about  actual  SUAS  employment 

Multiple: 

Just  as  good  & 

Better 

Just  as  good 

3.  Experiencing  actual  situations  that  might  arise  during  the 
employment  of  an  SUAS 

Better 

Just  as  good 

4.  Applying  tactical  knowledge  and  skills  to  the  employment 
of  SUAS 

Better 

Just  as  good 

5.  Identifying  gaps  in  your  knowledge  about  employment  of 
SUAS 

Better 

Just  as  good 

6.  Which  of  the  following  best  describes  how  you  felt  about 
the  initial  instruction  about  SUAS  employment? 

I  was  unfamiliar  with  a 

lot  of  the  material  and 
really  found  it 
informative 

I  was  unfamiliar  with  a 

lot  of  the  material  and 
really  found  it 
informative 

7.  The  instruction  I  was  presented  with  held  my  attention. 

Multiple:  Neutral  & 
Strongly  Agree  & 
Strongly  Disagree 

Multiple:  Neutral  & 
Strongly  Disagree 

8. 1  felt  that  the  instruction  provided  was  helpful  in  my 
subsequent  ability  to  complete  the  scenario(s). 

Agree 

Agree 

9.  Which  of  the  following  best  described  how  you  felt  about 
the  scenario(s)  you  completed? 

It  was  the  right  level  of 
challenge  for  me 

It  was  the  right  level  of 
challenge  for  me 

10.  The  scenario(s)  I  worked  on  were  realistic 

Agree 

Multiple:  Neutral  & 

Agree 

11.  The  scenario(s)  I  worked  on  held  my  attention 

Agree 

Agree 

12.  The  feedback  given  during  the  scenarios  was  helpful 

Agree 

Agree 

13.  The  hints  available  during  the  scenarios  were  helpful 

Agree 

Neutral 

14. 1  think  this  would  make  a  good  training  system  for  leaders 
at  company  and  below  with  organic  SUAS 

Agree 

Multiple:  Neutral  & 
Agree  &  Strongly 
Agree 
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SESSION  LOG  SUMMARIES 

■A  indicates  scenario  was  completed  successfully 
*  indicates  that  scenario  was  not  completed  successfully 
E  in  Soldier  ID  indicates  NCO 
O  in  Soldier  ID  indicates  officer 

Steps  during  scenario  attempts  represent  the  number  of  decisions  made 
min  =  minutes 


Adaptive  Condition 


Soldier  ID 

El 

E2 

03 

04 

Pretest  -  %  correct  (min) 

45.4%  (15.7) 

36.3%  (9.8) 

63.6%  (18.9) 

45.4%o  (9.3) 

Upfront  Instruction  -  #  of  ELOs 
(min) 

11  (39.5) 

10(17.9) 

7(13.4) 

11  (22.5) 

Offense  Scenario  Tty  1  -  steps  (min) 

%12  (12.2) 

2(2.1) 

%12  (16.2) 

%  12  (11.8) 

Remediation  -  #  of  ELOs  (min) 

4(1.0) 

Offense  Scenario  Try  2  -  steps  (min) 

5  (1.2) 

Remediation  -  #  of  ELOs  (min) 

2  (0.6) 

Offense  Scenario  Tty  3  -  steps  (min) 

*4(0.3) 

Remediation  -  #  of  ELOs  (min) 

3(1.0) 

Defense  Scenario  Tty  1  -  steps  (min) 

%11  (15.8) 

1  (0.3) 

%11  (18.27) 

%  11  (12.5) 

Remediation  -  #  of  ELOs  (min) 

2  (0.6) 

Defense  Scenario  Tty  2  -  steps  (min) 

7(0.9) 

Remediation  -  #  of  ELOs  (min) 

1  (0.3) 

Defense  Scenario  Tty  3  -  steps  (min) 

*2(0.3) 

Remediation  -  #  of  ELOs  (min) 

3  (0.6) 

Posttest  -  %  correct  (min) 

54.5%  (12) 

63.6%  (9.5) 

72.7%  (13.8) 

81.8%  (5.7) 
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Soldier  ID 

E6 

07 

08 

09 

Pretest  -  %  correct  (min) 

54.5%  (13.8) 

59.1%  (10.2) 

59.0%  (8.2) 

54.5%o 

(13.41) 

Upfront  Instruction  -  #of  ELOs 
(min) 

12  (24.1) 

10(38.1) 

10  (11.5) 

Offense  Scenario  Try  1  -  steps  (min) 

6(5.1) 

11  (28.2) 

11  (14.8) 

P' 11  (50.4) 

Remediation  -  #  of  ELOs  (min) 

1(1.6) 

1(1.6) 

1  (0.4) 

Offense  Scenario  Tty  2  -  steps  (min) 

p'  5  (1.9) 

P'4  (1.3) 

5  (2.3) 

Remediation  -  #  of  ELOs  (min) 

2  (0.5) 

Offense  Scenario  Tty  3  -  steps  (min) 

' 5(1.2 ) 

Remediation  -  #  of  ELOs  (min) 

Defense  Scenario  Tty  1  -  steps  (min) 

4(2.65) 

11(4.7) 

P' 11  (6.4) 

Sll  (18.7) 

Remediation  -  #  of  ELOs  (min) 

2  (4.1) 

4(2.0) 

Defense  Scenario  Tty  2  -  steps  (min) 

4  (.9) 

4  (6.3) 

Remediation  -  #  of  ELOs  (min) 

2  (.8) 

3(1.1) 

Defense  Scenario  Tty  3  -  steps  (min) 

*4  (.8) 

S4  (1.5) 

Remediation  -  #  of  ELOs  (min) 

2  (3.18) 

Posttest  -  %  correct  (min) 

36.4%  (6.6) 

86.4%  (6.8) 

81.8%  (4.6) 

50  %>  (8.2) 

Soldier  ID 

Oil 

012 

E13 

013 

Pretest  -  %>  correct  (min) 

63.6%  (8.9) 

50%  (31.6) 

45.4  %> 
(19.6) 

45.4%  (11.4) 

Upfront  Instruction  -  #  of  ELOs 
(min) 

7(28.3) 

10(48.6) 

10(8.6) 

9(46.4) 

Offense  Scenario  Tty  1  -  steps  (min) 

S 12  (20.3) 

7(26.7) 

P'11  (22.0) 

S12  (12.8) 

Remediation  -  #  of  ELOs  (min) 

1  (1.2) 

Offense  Scenario  Try  2  -  steps  (min) 

4  (2.4) 

Remediation  -  #  of  ELOs  (min) 

1(0.6) 

Offense  Scenario  Tty  3  -  steps  (min) 

S4  (2.2) 

Remediation  -  #  of  ELOs  (min) 

Defense  Scenario  Tty  1  -  steps  (min) 

P'1/ (7.3) 

11  (8.0) 

6(8.0) 

11  (7.6) 

Remediation  -  #  of  ELOs  (min) 

2  (0.9) 

1  (.63) 

1  (0.2) 

Defense  Scenario  Tty  2  -  steps  (min) 

4(2.9) 

6  (3.8) 

P' 4(1.0 ) 

Remediation  -  #  of  ELOs  (min) 

1  (0.6) 

1  (1.9) 

Defense  Scenario  Tty  3  -  steps  (min) 

*4(1.1) 

*3(3.6) 

Remediation  -  #  of  ELOs  (min) 

1  (0.4) 

2  (0.3) 

Posttest  -  %  correct  (min) 

72.7  %  (1.9) 

68%  (5.4) 

45.4  %>  (7.0) 

59.1%  (4.1) 

F-2 


Soldier  ID 

E5 

Pretest  -  %  correct  (min) 

31.8%  (15.4) 

Upfront  Instruction  -  #  of  ELOs 
(min) 

9(4.7) 

Offense  Scenario  Tty  1  -  steps  (min) 

Sll  (41.2) 

Remediation  -  #  of  ELOs  (min) 

Offense  Scenario  Tty  2  -  steps  (min) 

Remediation  -  #  of  ELOs  (min) 

Offense  Scenario  Tty  3  -  steps  (min) 

Remediation  -  #  of  ELOs  (min) 

Defense  Scenario  Tty  1  -  steps  (min) 

4  (8.2) 

Remediation  -  #  of  ELOs  (min) 

1  (1.4) 

Defense  Scenario  Tty  2  -  steps  (min) 

7(7.5) 

Remediation  -  #  of  ELOs  (min) 

1  (1.6) 

Defense  Scenario  Tty  3  -  steps  (min) 

*3(2.7) 

Remediation  -  #  of  ELOs  (min) 

1  (0.4) 

Posttest  -  %  correct  (min) 

40.9%  (8.6) 

Nonadaptive  Condition 


Soldier 

Pretest 

Upfront 

Offense 

Defense 

Posttest 

ID 

%>  correct 

Instruction 

Scenario  - 

Scenario  - 

%>  correct 

(minutes) 

minutes 

minutes 

minutes 

(minutes) 

EC  10 

59.1%  (7.3) 

18.0 

S  19.7 

S7.9 

68.2%  (4.6) 

EC2 

36.4%)  (7.0) 

35.6 

*  2.6 

*  2.0 

36.6%  (6.6) 

EC3 

63.6%  (12.0) 

9.2 

S  7.6 

*  4.9 

59.1%  (5.7) 

EC4 

40.9%  (10.1) 

67.6 

S  2.7 

*  7.8 

68.2%  (5.6) 

EC5 

45.4%  (12.0) 

34.5 

S  19.0 

S11.0 

59.1%  (6.8) 

EC6 

31.8%  (12.3) 

38.6 

S  11.8 

*  7.2 

63.6%  (9.7) 

EC9 

27.3%  (10.3) 

92.6 

S  8.7 

*6.1 

77.3%  (5.6) 

0C1 

40.9%  (9.8) 

10.9 

S  8.2 

S  4.5 

50.0%  (7.2) 

0C11 

50.0%  (7.9) 

38.6 

'C  11.6 

S  6.2 

59.1%  (11.8) 

0C7 

54.5%  (6.3) 

51.4 

'C  15.8 

S  4.7 

68.2%  (4.3) 

0C8 

59.1%  (11.7) 

13.2 

S  1.0 

S  0.9 

59.1%  (6.3) 

F-3 


Appendix  G 


ACRONYMS 


BN 

Battalion 

CO 

Company  Commander 

ELO 

Enabling  Learning  Objective 

IMI 

Interactive  Multimedia  Instruction 

LMS 

Learning  Management  System 

METT-TC 

Mission,  Enemy,  Time,  Terrain/Weather,  Troops 
Available,  and  Civil 

NCO 

Noncommissioned  Officer 

OBJ 

Objective 

PL 

Platoon  Leader 

SUAS 

Small  Unmanned  Aerial  System 

TLO 

Terminal  Learning  Objective 

TTPs 

Tactics,  Techniques,  and  Procedures 

XO 

Executive  Officer 

G-1 


